JP2008193514A - Imaging apparatus, and object tracking method and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To track an object with high accuracy even though the object moves. <P>SOLUTION: A CPU 24 sets a tracking area within an imaging area, and tracks the object 2 placed in accordance with the tracking area. The CPU 24 supplies a timing signal S5 to a flash control part 14, an imaging control part 15, and element driving parts 102 and 103 of a sensor part 11 to drive the respective parts. The CPU 24 compares a color image acquired from a data processing part 20 with a color image acquired from a nonvolatile memory, detects a change in a position of the object 2 and acquires position data after the movement of the object 2. The CPU 24 compares a distance image acquired from the data processing part 20 with a distance image acquired from the nonvolatile memory, detects a change in a position of the object 2 and acquires position data after the movement. The CPU 24 selects the position data with the highest reliability and moves the tracking area on the basis of the selected position data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, a subject tracking method thereof, and a program.

  There is an imaging device including an autofocus device that performs a specific process such as focusing (focusing) by tracking a subject (see, for example, Patent Documents 1 and 2).

  The imaging apparatus described in Patent Document 1 stores an image of a subject to be focused as a reference pattern, and detects an image corresponding to the reference pattern from captured images by pattern matching, thereby changing the position of the subject. Is detected and the subject is tracked.

The imaging apparatus described in Patent Literature 2 detects a change in the position of the subject by detecting a change in entropy by dividing the focus area into a plurality of areas and calculating the entropy of the image for each area. The subject moving in the attention area is tracked.
JP 2006-58431 A (page 4, FIG. 1) Japanese Patent Laid-Open No. 5-80248 (page 3, FIG. 1)

  However, when the subject is tracked, various situations can be expected as the orientation of the subject changes and the shape changes, or another object changes in front of the subject.

  In the imaging apparatus described in Patent Document 1, when the orientation of the subject changes and the shape changes, pattern matching between the subject and the reference pattern cannot be performed, and the subject cannot be tracked.

  Further, in the imaging apparatus described in Patent Document 1, when the position of the subject changes back and forth with respect to the imaging apparatus, neither the shape nor the position of the reference pattern changes, and the change in the position of the subject cannot be detected.

  Further, in the imaging apparatus described in Patent Document 2, when an object having the same entropy as that of the subject crosses the front and rear of the subject, the object is tracked, and the subject may not be tracked.

  The present invention has been made in view of such a conventional problem, and an object thereof is to provide an imaging apparatus capable of following a subject with higher accuracy, a subject tracking method, and a program thereof.

In order to achieve this object, an imaging apparatus according to the first aspect of the present invention provides:
An image sensor that repeatedly images a subject;
A first image acquisition unit for acquiring a first image representing the shape of the subject including visual information of the subject obtained visually from the image of the subject captured by the imaging device;
A second image acquisition unit that acquires a second image representing the shape of the subject including distance information to the subject from the image of the subject captured by the imaging device;
Each time the image sensor performs imaging, the position of the subject is changed based on the first image acquired by the first image acquisition unit or the second image acquired by the second image acquisition unit. A position change detection unit to detect;
A tracking region moving unit that tracks the subject and moves a tracking region for performing specific control when the position change detection unit detects a change in the position of the subject. .

  The tracking area moving unit may perform autofocus control as the specific control, and continue to acquire images while performing autofocus control on the tracking area.

  The tracking area moving unit may perform superimposing display control of a predetermined image as the specific control, and continue to display the predetermined image superimposed on the tracking area.

  The follow-up area moving unit may perform auto-framing control as the specific control so as to keep following the frame corresponding to the follow-up area.

A storage unit that stores the first image acquired by the first image acquisition unit and the second image acquired by the second image acquisition unit each time the imaging element performs imaging;
The position change detection unit compares the first image stored in the storage unit with the first image newly acquired by the first image acquisition unit, and is stored in the storage unit. By comparing the second image and the second image newly acquired by the second image acquisition unit, a change in the position of the subject is detected based on one of the comparison results. May be.

  The position change detector is configured to acquire the first object newly acquired by the subject and the first image acquisition unit included in the first image stored in the storage unit each time the image sensor performs imaging. A change in the position of the subject is detected in accordance with a first analysis method that compares the subject included in the two images and analyzes the positional relationship of the subject included in each of the two compared first images by pattern matching. You may make it do.

  The position change detection unit compares the first image stored in the storage unit with the first image newly acquired by the first image acquisition unit every time the image sensor performs imaging. Then, a second analysis for analyzing the positional relationship of the subject included in the two first images compared by obtaining the entropy of each of the divided small areas by dividing the imaging area into a plurality of small areas. A change in the position of the subject may be detected according to a method.

  The position change detector is configured to acquire the first object newly acquired by the subject and the first image acquisition unit included in the first image stored in the storage unit each time the image sensor performs imaging. The subject included in the first image is included in the two first images compared by comparing the subject included in the image and obtaining the color difference signal of each small region divided by dividing the imaging region into a plurality of small regions. The change in the position of the subject may be detected in accordance with a third analysis method for analyzing the positional relationship of the subject.

  The position change detection unit acquires the first image or the second image and the first image acquisition unit or the second image acquisition stored in the storage unit every time the image sensor performs imaging. And the first image stored in the storage unit and the first image when another object crossing the front and rear of the subject is detected by comparing the image with the newly acquired image by the unit The acquisition unit compares the subject included in the first image newly acquired, selects the third analysis method, and detects a change in the position of the subject according to the selected third analysis method. You may do it.

  The position change detection unit compares the second image stored in the storage unit with the second image newly acquired by the second image acquisition unit every time the image sensor performs imaging. Then, a change in the position of the subject may be detected according to a first analysis method in which the positional relationship between the subjects included in the two compared second images is analyzed by pattern matching.

  The position change detection unit compares the second image stored in the storage unit with the second image newly acquired by the second image acquisition unit every time the image sensor performs imaging. Then, a second analysis for analyzing the positional relationship of the subject included in the two second images compared by acquiring the entropy of each of the divided small areas by dividing the imaging area into a plurality of small areas. A change in the position of the subject may be detected according to a method.

  The position change detection unit acquires the first image or the second image and the first image acquisition unit or the second image acquisition stored in the storage unit every time the image sensor performs imaging. The second image stored in the storage unit and the second image according to the second analysis method when a change in the shape of the subject is detected by comparing the newly acquired image with the image The change in the position of the subject may be detected based on a comparison result with the second image newly acquired by the acquisition unit.

Each time the position change detection unit captures an image,
Each time the first image acquisition unit and the second image acquisition unit acquire the first image and the second image, respectively, based on the position data of the plurality of subjects stored in the storage unit. Set the range of movement direction of the subject,
The position of the subject included in the first image newly acquired by the first image acquisition unit, and the position of the subject included in the second image newly acquired by the second image acquisition unit When the position is outside the set moving direction, the newly acquired position data of the subject is discarded, and only when the position is within the determined moving direction, the newly acquired position data of the subject is stored in the storage unit. You may make it memorize.

The image sensor receives light reflected by the subject, and includes a first pixel unit for acquiring the first image and a second pixel unit for acquiring the second image. As the pixel unit, a plurality of pixel units are arranged in a matrix.
A first element driving unit that drives the plurality of first pixel units so as to acquire the first image;
A second element driving unit that drives the plurality of second pixel units so as to acquire the second image.

Each of the first pixel units includes a plurality of light receiving units that receive the light reflected by the subject for each color component to generate a signal charge, and output the signal charge generated for each color component. ,
The first element driving unit may drive the plurality of light receiving units for each of the color components.

The second image acquisition unit includes:
A light projecting unit that projects light;
An imaging control unit that blinks the light projected by the light projecting unit;
A plurality of second pixel portions of the image sensor;
A second element driving unit for driving the plurality of second pixel units;
A calculation unit that obtains distance information to the subject based on the signal charges output from each of the second pixel units,
Each of the second pixel units generates a signal charge by receiving the flashing light projected from the light projecting unit and reflected by the subject, and the generated signal charge is a period during which the light projecting unit is lit. The first signal charge generated by the reflected light received in the light and the second signal charge generated by the reflected light received during the period when the light projecting part is turned off are output separately.
Based on the amount of the first signal charge and the amount of the second signal charge output from each second pixel unit, the arithmetic unit turns on the light projecting unit and then turns on the light receiving unit. A delay time until the reflected light is received may be measured, and distance information to the subject may be acquired based on the measured delay time.

An object tracking method for an imaging apparatus according to a second aspect of the present invention includes:
Causing the image sensor to repeatedly image the subject;
Obtaining a first image representing the shape of the subject including visual information of the subject obtained visually from an image captured each time the image sensor performs imaging;
Obtaining a second image representing the shape of the subject, including distance information to the subject, from an image captured each time the image sensor performs imaging;
Detecting a change in the position of the subject based on the first image or the second image acquired each time the image sensor performs imaging;
And a step of moving a follow-up area for performing specific control, which is set to track the subject when a change in the position of the subject is detected.

The program according to the third aspect of the present invention is:
On the computer,
A procedure for repeatedly imaging a subject with an image sensor;
A procedure for acquiring a first image representing the shape of the subject including visual information of the subject obtained visually from an image captured each time the image sensor performs imaging,
A procedure for acquiring a second image representing the shape of the subject including distance information to the subject from an image captured each time the image sensor performs imaging,
A procedure for detecting a change in the position of the subject based on the first image or the second image acquired each time the subject is photographed;
A procedure for moving a tracking area for performing specific control, which is set to track the subject when detecting a change in the position of the subject;
Is to execute.

  According to the present invention, it is possible to accurately follow a subject.

Hereinafter, an apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 shows the configuration of the imaging apparatus according to this embodiment.
The imaging device 1 according to the present embodiment includes a sensor unit 11, flashes 12 and 13, a flash control unit 14, an imaging control unit 15, an imaging lens 16, a lens driving unit 17, a frame memory 18, and a memory. A controller 19, a data processing unit 20, a ROM 21, a RAM 22, a nonvolatile memory 23, and a CPU 24 are provided.

  The imaging device 1 irradiates the subject 2 with visible light and near infrared light with the flashes 12 and 13, respectively, and receives the reflected light reflected by the subject 2 within the imaging area 3 of the sensor unit 11.

  As shown in FIG. 2, when the subject 2 is three subjects 2A, 2B, and 2C in which characters are drawn and positions are different as shown in FIG. In the imaging area 3, color images 2Ai, 2Bi, and 2Ci are acquired, respectively.

  The color images 2Ai, 2Bi, and 2Ci are images representing the shapes of the subjects 2A, 2B, and 2C including visual information obtained visually, for example, RGB components and luminance information of the subjects 2A, 2B, and 2C. .

  In addition, the imaging apparatus 1 acquires distance images 2Ai, 2Bi, and 2Ci within the imaging area 3, as shown in FIG. The distance images 2Ad, 2Bd, and 2Cd are images representing the shapes of the subjects 2A, 2B, and 2C including distance information from the imaging device 1 to the subjects 2A, 2B, and 2C, respectively.

  Assuming that the distance between the imaging device 1 and the subjects 2A, 2B, and 2C is 2A <2B <2C, the imaging device 1 sets the brightness of the distance images 2Ad, 2Bd, and 2Cd to 2Ad as shown in FIG. It is set so that> 2Bd> 2Cd.

  The imaging device 1 of the present embodiment is configured to acquire the color images 2Ai, 2Bi, 2Ci and the distance images 2Ad, 2Bd, 2Cd at the same time.

  The imaging apparatus 1 also has a function of tracking the subject 2 and performing AF processing so that the subject 2 is focused.

  Returning to FIG. 1, the sensor unit 11 includes an imaging element 101 and element driving units 102 and 103.

  The image sensor 101 acquires a color image and a distance image, and includes a pixel circuit 101G, a pixel circuit 101R, a pixel circuit 101B, and a pixel circuit 101L as shown in FIG.

  The pixel circuits 101G, 101R, and 101B are for acquiring the G (Green) component, R (Red) component, and B (Blue) component of the subject 2, respectively. The pixel circuit 101L is for acquiring a distance image.

  The image sensor 101 is composed of a plurality of pixel units, with one pixel circuit 101G, 101R, 101B, 101L as one pixel unit. In the present embodiment, it is assumed that the pixel units are arranged in a matrix of n rows × n columns (n is a natural number excluding 0).

  The n × n pixel circuits 101G, 101R, 101B, and 101L constitute G, R, and B component color image acquisition pixel arrays, respectively, and the n × n pixel circuit 101L acquires distance images. A pixel array is constructed.

  The pixel circuits 101G, 101R, 101B, and 101L arranged in a matrix of the pixel array are respectively connected to the pixel circuits 101G (i, j), 101R (i, j), 101B (i, j), and 101L. It is expressed as (i, j) (i, j; integer, 1 ≦ i ≦ n, 1 ≦ j ≦ n).

  The imaging device 1 includes a color image acquisition unit and a distance image acquisition unit configured to include such a pixel array. The color image acquisition unit is for acquiring color images 2Ai, 2Bi, 2Ci as shown in FIG. 3A, and the distance image acquisition unit is a distance image 2Ad, as shown in FIG. This is for acquiring 2Bd and 2Cd.

  FIG. 5 is a diagram illustrating a configuration of a color image acquisition unit 200 configured to include a pixel array for acquiring a G component color image. The color image acquisition unit 200 includes a flash 12, a flash control unit 14, a pixel array 201, a vertical scanning unit 202, a horizontal scanning unit 203, transistors Q1-1 to Q1-n, an amplifier 204, an A / A And a D converter 205.

  The imaging apparatus 1 includes the R component and B color image acquisition units 200 in addition to the G component color image acquisition unit 200 having the above-described configuration. In the present embodiment, only the color image acquisition unit 200 for the G component will be described.

  1 includes a vertical scanning unit 202 for G, R, and B components, a horizontal scanning unit 203, transistors Q11-1 to Q11-n, and an amplifier 204. And a converter 205.

  The flash 12 emits visible light toward the subject 2. The flash controller 14 is supplied with a timing signal St from the CPU 24 and causes the flash 12 to emit light.

  The pixel array 201 receives visible light reflected by the subject 2 and is configured by the n × n pixel circuits 101G (i, j) as described above. As shown in FIG. 6, each pixel circuit 101G (i, j) includes a photodiode PD11 and transistors Q11 to Q13.

  The photodiode PD11 receives the G component light reflected by the subject 2 and generates and outputs a signal charge corresponding to the amount of received light, and the anode is grounded.

  The transistor Q11 is a transistor that resets the photodiode PD11 when the reset signal R11 is supplied to the gate (terminal).

  The gate of the transistor Q11 is connected to the reset line L11, the source is connected to the cathode of the photodiode PD11, and a positive voltage is applied to the drain.

  The transistor Q12 is for amplifying the current due to the signal charge generated by the photodiode PD11, its gate is connected to the cathode of the photodiode PD11, and a positive voltage is applied to its drain.

  The transistor Q13 is a row selection transistor, the gate is connected to the row selection line L12, the drain is connected to the source of the transistor Q12, and the source is connected to the vertical signal line L13.

  The vertical scanning unit 202 is supplied with a timing signal St from the CPU 24, selects a row, resets the pixel circuit 101G in the row, and controls signal output. The vertical scanning unit 202 resets the pixel circuit 101G of the selected row by outputting a reset signal R11 to the gate of the transistor Q11 of the pixel circuit 101G via the reset line L11 of the selected row.

  The vertical scanning unit 202 controls the signal output by outputting the row selection signal S11 to the gate of the transistor Q13 of each pixel circuit 101G via the row selection line L12 of the selected row.

  The transistors Q1-1 to Q1-n are for controlling the output of the signal Si from the pixel circuit 101G (i, j) connected to the row selection line L12, and each drain is connected to each vertical signal line L13. And each source is connected to the input of an amplifier 204.

  The horizontal scanning unit 203 is supplied with a timing signal St from the CPU 24, controls the transistors Q1-1 to Q1-n, and selects and outputs the signal Si.

  The horizontal scanning unit 203 outputs a high-level column selection signal S12 to the gate of the transistor Q1-1 in the first column to turn on the transistor Q1-1. When the transistor Q1-1 outputs the signal Si, the signal level of the column selection signal S12 is set to a low level, and the transistor Q1-1 is turned off.

  The horizontal scanning unit 203 sequentially outputs the column selection signal S12 to the second column transistor Q1-2,..., The nth column transistor Q1-n, and performs such processing.

  The amplifier 204 amplifies the signal voltage of the signal Si output from each transistor Q1-1 to Q1-n.

  The A / D converter 205 converts the analog signal Si obtained by amplifying the signal voltage by the amplifier 204 into digital data as color image data. The element driving unit 102 outputs the color image data converted by the A / D converter 205.

  Next, the configuration of the distance image acquisition unit 300 configured to include a pixel array for acquiring a distance image will be described. As the distance image acquisition unit 300, for example, one described in Japanese Patent Application Laid-Open No. 2004-294420 is used.

  The distance image acquisition unit 300 irradiates the subject 2 with pulsed light, and acquires a distance image based on the flight time of light until the pulsed light is reflected by the subject 2 and received.

  As shown in FIG. 7, the distance image acquisition unit 300 includes a flash 13, an imaging control unit 15, a pixel array 301, a light receiving lens 302, a timing control unit 303, a vertical shift register 304, and a sample hold unit. 305-1 to 305-n, switch units 306-1 to 306-n, a horizontal shift register 307, output buffers 308-1 and 308-2, a calculation unit 309, an A / D converter 310, Consists of.

  1 includes a timing control unit 303, a vertical shift register 304, sample hold units 305-1 to 305-n, and switch units 306-1 to 306-n. , A horizontal shift register 307, output buffers 308-1 and 308-2, an arithmetic unit 309, and an A / D converter 310.

  The flash 13 irradiates near-infrared light toward the subject 2. The imaging controller 15 is instructed by the CPU 24 to acquire the distance image and controls the flash 13 and the sensor unit 11.

  The imaging control unit 15 includes a clock signal generation unit (not shown) that generates the clock signal CLK as shown in FIG. 8A and a counter that counts the number of clocks of the clock signal CLK.

  From time t0, the imaging control unit 15 performs pulse modulation of near-infrared light by blinking the flash 13 every time the level of the clock signal CLK generated by the clock signal generation unit is inverted.

  Note that the period Tp indicates a period in which the signal level of the clock signal CLK is high, the period Tq indicates the period in which the signal level of the clock signal CLK is low, and the period Tp and the period Tq are equal. And

  In order to detect the dark current of the photodiode PD11 in the pixel circuit 101L, the imaging control unit 15 turns off the flash 13 at time t1 when the count number is N (N is a natural number equal to or greater than 1). A period Ta (time t0 to t1) illustrated in FIG. 8 indicates a period in which the flash 13 is blinked to perform pulse modulation.

  Then, the counter resets the count number and again counts the clock number of the clock signal CLK from time t1, and the imaging control unit 15 turns off the flash 13 until time t2 when the clock number becomes N. A period Tb (time t1 to t2) shown in FIG. 8 indicates a period during which the flash 13 is turned off.

  The imaging control unit 15 supplies the clock signal CLK generated by the clock signal generation unit to the timing control unit 303 and the vertical shift register 304.

  The pixel array 301 receives the flash light reflected from the subject 2 by the near-infrared light irradiated by the flash 13, and is configured by the n × n pixel circuit 101L (i, j) as described above. .

  The distance image acquisition unit 300 according to the present embodiment has a CMOS type configuration. FIG. 9A is a diagram showing a configuration of each pixel circuit 101L (i, j) according to the present embodiment, and FIG. 9B is an equivalent circuit of each pixel circuit 101L (i, j). It is a figure which shows a structure.

As shown in FIG. 9A, each pixel circuit 101L (i, j) according to this embodiment includes an n layer 311a, a p ⁺ layer 311b, an n ⁺ layer 311c, 311d, a silicon oxide film 311e, and a control electrode 311f. 311g, a light shielding film 312, transistors Q21 and Q22, and an amplifier 313.

The photodiode PD12 shown in FIG. 9B receives the light reflected by the subject 2 and generates and outputs a signal charge corresponding to the amount of the received light. N layer 311a and p ⁺ layer 311b shown in FIG.

The transistor Q21 shown in FIG. 9B includes the p-type silicon substrate 311, the n layer 311a, the control electrode 311f, and the n ⁺ layer 311c shown in FIG. 9A. In this case, the n layer 311a, the n ⁺ layer 311c, and the control electrode 311f serve as the drain, source, and gate of the transistor Q21, respectively.

The transistor Q22 shown in FIG. 9B includes the p-type silicon substrate 311, the n layer 311a, the control electrode 311g, and the n ⁺ layer 311d shown in FIG. 9A. In this case, the n layer 311a, the n ⁺ layer 311d, and the control electrode 311g serve as the drain, source, and gate of the transistor Q22, respectively.

  The silicon oxide film 311 e is for insulating the p-type silicon substrate 311 and the layers formed on the p-type silicon substrate 311, and transmits near-infrared light reflected from the subject 2.

The light shielding film 312 opens only above the n layer 311a and the p ⁺ layer 311b corresponding to the photodiode PD12, and shields the p-type silicon substrate 311 excluding this region. Thereby, only the photodiode PD12 receives near-infrared light reflected from the subject 2.

  The control electrodes 311f and 311g are electrodes to which control signals TX1 and TX2 are applied, respectively. When a high-level control signal TX1 is applied to the control electrode 311f, a channel is formed under the control electrode 311f, and the transistor Q21 shown in FIG. 9B is turned on.

  Similarly, when a high level control signal TX2 is applied to the control electrode 311g, a channel is formed under the control electrode 311g, and the transistor Q22 shown in FIG. 9B is turned on.

  Therefore, when the high level control signals TX1 and TX2 are alternately applied to the control electrodes 311f and 311g, respectively, and the transistor Q21 is turned on, the current due to the signal charge accumulated in the photodiode PD12 is the drain-source of the transistor Q21. To the vertical signal line L21.

  On the other hand, when the transistor Q22 is turned on, a current due to the signal charge accumulated in the photodiode PD12 flows to the vertical signal line L22 via the drain-source of the transistor Q22.

  In this manner, each pixel circuit 101L is supplied with the high-level control signals TX1 and TX2 alternately, and thereby the first signal charge generated by the reflected light received during the period Tp during which the flash 13 is on and The second signal charge generated by the reflected light received during the period Tq during which the flash 13 is turned off is output separately.

  Transistors Q23 and Q24 shown in FIGS. 9A and 9B are for resetting the photodiode PD12.

  The sources of the transistors Q23 and Q24 are respectively connected to the n + layers 311c and 321d shown in FIG. 9A, and currents are supplied to the respective drains from the power supply line of voltage + V.

  The amplifier 313 is for amplifying the output currents of the transistors Q21 and Q22 shown in FIG. 9B, and includes transistors Q25 to Q27. A current is supplied to the drain of the transistor Q25 from the power supply line of the voltage + V.

  The drains of the transistors Q26 and Q27 are connected to the source of the transistor Q25, and the sources are connected to the vertical signal lines L21 and L22, respectively. The gates of transistors Q26 and Q27 are connected to the sources of transistors Q21 and Q22, respectively.

  The transistor Q25 is turned on when a high-level row selection signal S21 is supplied to the gate, and supplies current to the transistors Q26 and Q27. When the transistor Q21 supplies current to the gate of the transistor Q26, the gate voltage becomes high level and the transistor Q26 is turned on.

  The transistor Q26 is turned on, amplifies the output current of the transistor Q21, outputs the signal Ad as a signal Sd1, and outputs the signal current of the signal Sd1 to the vertical signal line L21.

  When the transistor Q22 supplies current to the gate of the transistor Q27, the gate voltage of the transistor Q27 becomes high level and the transistor Q27 is turned on.

  The transistor Q27 turns on, amplifies the output current of the transistor Q22, outputs the signal current of the signal Sd2 to the vertical signal line L22 as the signal Sd2.

  The light receiving lens 302 shown in FIG. 7 is a lens for condensing light on the photodiode PD12 of each pixel circuit 101L (i, j).

  The timing control unit 303 controls the output timing of outputting the signal currents of the signals Sd1 and Sd2 from each pixel circuit 101L (i, j) in synchronization with the clock timing of the clock signal CLK supplied from the imaging control unit 15. Is for.

  The timing control unit 303 supplies the control signal TX1 for each row to each pixel circuit 101L (i, j) via each control line L23. As shown in FIG. 8D, the timing control unit 303 sets the signal level of the control signal TX1 to a high level in the period Tp and to a low level in the period Tq.

  In addition, the timing control unit 303 supplies the control signal TX2 for each row to each pixel circuit 101L (i, j) via each control line L24. As shown in FIG. 8E, the timing control unit 303 sets the signal level of the control signal TX2 to a low level in the period Tp and to a high level in the period Tq.

  The vertical shift register 304 is for selecting a row, resetting the pixel circuit 101L (i, 1) in the selected row, and selecting the pixel circuit 101L in the row that outputs the signal currents of the signals Sd1 and Sd2. It is.

  When the vertical shift register 304 selects the pixel circuits 101L (1, j) to 101L (n, j) in the j-th row, as shown in FIG. 8F, the clocks at the beginning of the period Ta and the period Tb are displayed. In one cycle of the signal CLK, a high-level reset signal R21 is supplied to the pixel circuit 101L (i, 1) via the reset line L15 shown in FIG.

  Further, the vertical shift register 304 sets the reset signal R21 to the low level and ends the reset when one cycle of the clock signal CLK in each of the periods Ta and Tb ends.

  Then, as shown in FIG. 8G, the vertical shift register 304 selects the high-level row in the pixel circuits 101L (1, j) to 101L (n, j) until the periods Ta and Tb are completed. A signal S21 is supplied.

  The sample hold units 305-1 to 305-n output the signal current of the signal Sd1 output from each pixel circuit 101L (i, j) via the vertical signal line L21 to the vertical signal line L22 for each column. The signal current of the signal Sd2 is individually stored, and each signal voltage is sampled and held. The sample hold units 305-1 to 305-n include sample hold circuits 321 and 322, respectively.

  The sample hold circuit 321 accumulates the signal current of the signal Sd1 output from each pixel circuit 101L (i, j) via the vertical signal line L21, and samples and holds the signal voltage based on the accumulated signal current.

  The sample hold circuit 322 accumulates the signal current of the signal Sd2 output from each pixel circuit 101L (i, j) via the vertical signal line L22, and samples and holds the signal voltage based on the accumulated signal current.

  The switch units 306-1 to 306-n control the output of the signal voltage of the signal Sd1 and the signal voltage of the signal Sd2 sampled and held by the sample hold circuits 321 and 322 of the sample hold units 305-1 to 305-n, respectively. Is for.

  The horizontal shift register 307 selects any one switch unit 306-j from the switch units 306-1 to 306-n and turns on the selected switch unit 306-i.

  As shown in FIG. 8H, the horizontal shift register 307 turns on the switch unit 306-i by outputting a high-level column selection signal S22 to the selected switch unit 306-i.

  The output buffers 308-1 and 308-2 are for temporarily storing individually the signal voltage of the signal Sd1 and the signal voltage of the signal Sd2 output from the sample hold units 305-1 to 305-n, respectively. .

  The calculation unit 309 calculates the distance to the subject 2 based on the signal voltage of the signal Sd1 and the signal voltage of the signal Sd2 temporarily stored in the output buffers 308-1 and 308-2, and acquires distance image data. To do.

  The arithmetic unit 309 includes a memory that stores the signal voltage temporarily stored in the output buffer 308-1. The arithmetic unit 309 then outputs the signal voltage V1 of the signal Sd1 and the signal voltage V2 of the signal Sd2 sampled and held by the sample hold circuits 321 and 322 of the sample hold unit, respectively, in the output buffer 308-1. Acquired from 308-2 and stored in the built-in memory.

  In addition, the calculation unit 309 acquires the signal voltage V1 ′ of the signal Sd1 and the signal voltage V2 ′ of the signal Sd2 sampled and held by the sample hold circuits 321 and 322 from the output buffer 308-1, respectively, during the period Tb. Stored in the built-in memory. The calculation unit 309 calculates the distance to the subject 2 based on the signal voltages V1, V1 ', V2, and V2' stored in the built-in memory, and acquires distance image data.

A process of acquiring distance image data of the calculation unit 309 will be described.
As shown in FIGS. 8B and 8C, the photodiode PD12 receives the flash light in the period Ta with a delay time Td after the flash 13 is turned on. The delay time Td corresponds to the flight time of light traveling back and forth between the imaging device 1 and the subject 2, and is proportional to the distance between the imaging device 1 and the subject 2.

  The photodiode PD12 of the pixel circuit 101L (i, j) generates signal charges while receiving flash light. With the received light intensity constant, the amount of signal charge accumulated in the photodiode PD12 by the received light is proportional to the received time.

The signal voltage V1 is expressed by the following equation (1).
V1 = Gc · N · (Id1 · (Tp−Td)
+ Id1 · 2Tp + Ib · Tp)
(1)
Where Gc: Gain
N: Number of pulses of the clock signal CLK
Ip: Photodiode PD12
Current (quantity) due to signal charge generated by
Id1: From photodiode PD12
Dark current flowing through transistor Q21

The signal voltage V2 is expressed by the following equation (2).
V2 = Gc · N · (Id1 · Td + Id2 · 2Tq + Ib · Tq)
(2)
However, from Id1: photodiode PD12
Dark current (current amount) flowing through transistor Q22

The signal voltage V1 ′ is expressed by the following equation (3).
V1 ′ = Gc · N · (Id1 · 2Tp + Ib · Tp) (3)

The signal voltage V2 ′ is expressed by the following equation (4).
V2 ′ = Gc · N · (Id2 · 2Tq + Ib · Tq) (4)

但し、
Ｖ１１＝Ｖ１＋Ｖ２
Ｖ１２＝Ｖ１’＋Ｖ２’
Ｖ１３＝Ｖ１−Ｖ１’
Ｖ１４＝Ｖ２−Ｖ２’
・・・・・・（５） Tp = Tq, and the delay time Td is expressed by the following equation (5) from the equations (1) to (4).

However,
V11 = V1 + V2
V12 = V1 '+ V2'
V13 = V1-V1 '
V14 = V2-V2 '
(5)

  The computing unit 309 computes the delay time Td based on the signal voltages V1 and V1 'of the signal Sd1 and the signal voltages V2 and V2' of the signal Sd2 according to the equation (5).

  The delay time Td is proportional to the flight time of light to the subject 2, and the flight time of light is proportional to the distance to the subject 2. Therefore, the calculation unit 309 calculates the distance to the subject 2 based on the delay time Td.

  The calculation unit 309 performs such calculation for each pixel circuit 101L (i, j) for each row and each column, and outputs the calculated distance value in analog form.

  As described above, the arithmetic unit 309 measures the delay time from when the flash 13 is turned on until the photodiode PD12 receives the reflected light, based on the amount of the two signal charges output from each pixel circuit 101L. Based on the measured delay time, distance information to the subject 2 is acquired.

  The A / D converter 310 converts the analog distance value output from the arithmetic unit 309 into digital data as distance image data. The element driving unit 103 outputs the distance image data converted by the A / D converter 310.

  As described above, each pixel circuit 101L is generated by the first signal charge generated by the reflected light received during the period Tp when the flash 13 is turned on and the reflected light received during the period Tq when the flash 13 is turned off. The second signal charge is output separately.

  Then, each pixel circuit 101L determines a delay time from when the flash 13 is turned on until the photodiode PD12 receives the reflected light based on the first signal charge amount and the second signal charge amount. Measurement is made, and distance information to the subject 2 is acquired based on the measured delay time.

  Returning to FIG. 1, the imaging lens 16 is used to form an image of the subject 2 on the imaging element 101. The lens driving unit 17 is for driving the imaging lens 16 so that the control signal Sc1 is supplied from the CPU 24 and the subject 2 is focused.

  The frame memory 18 stores color image data and distance image data output from the element driving unit 102 (A / D converter 204) and the element driving unit 103 (A / D converter 309) of the sensor unit 11, respectively. And has a color image area 18a and a distance image area 18b.

  The color image area 18a is an area for storing color image data output from the element driving unit 102 (amplifier 204) of the sensor unit 11, and the distance image area 18b is an element driving unit 103 (calculation) of the sensor unit 11. This is an area for storing the distance image data output from the unit 309).

  The memory controller 19 performs storage processing in the frame memory 18, and stores the color image data and the distance image data output from the sensor unit 11 in the color image area 18a and the distance image area 18b, respectively.

  The memory controller 19 is output from the element driving unit 102 (A / D converter 204) of the sensor unit 11 for each pixel circuit 101G (i, j), 101R (i, j), 101B (i, j). The obtained color image data is stored in the color image area 18a.

  Further, the memory controller 19 stores the distance image data output from the element driving unit 103 ((A / D converter 310) of the sensor unit 11 in the distance image region 18b for each pixel circuit 101L (i, j). .

  The data processing unit 20 performs data processing on the color image data and distance image data stored in the frame memory 18. The data processing unit 20 acquires G, R, and B component color images for each pixel based on the color image data stored in the color image area 18 a of the frame memory 18.

  Further, the data processing unit 20 sets the color and brightness for each pixel based on the distance image data stored in the distance image area 18b of the frame memory 18, and acquires a distance image.

  The ROM 21 is a memory for storing a program executed by the CPU 24. The RAM 22 is for supplying data necessary for the CPU 24 to execute processing. The nonvolatile memory 23 stores color image and distance image data acquired by the data processing unit 20.

  The CPU 24 controls the entire image pickup apparatus 1 according to a program stored in the ROM 21.

  Specifically, for example, when the imaging apparatus 1 is provided in a digital camera, when the half shutter is pressed, the CPU 24 reads the AF processing program from the ROM 21 and executes the AF processing. As this process, the CPU 24 first sets a tracking area (focus area) FA in the imaging area 3 as shown in FIG.

  The CPU 24 repeatedly supplies the control signal Sc1 to the lens driving unit 17 at a preset period, adjusts the focal position of the imaging lens 16, and focuses the subject 2 adjusted to the tracking area FA.

  The CPU 24 focuses the subject 2 by, for example, a hill climbing servo system. This hill-climbing servo method is a method of acquiring a specific high-frequency component from a signal obtained from the image sensor 101 and moving the position of the imaging lens 16 back and forth so that the amplitude of this high-frequency component is maximized.

  The CPU 24 repeatedly supplies the timing signal St to the flash control unit 14, the imaging control unit 15, and the element driving units 102 and 103 of the sensor unit 11 in a preset cycle to operate each unit.

  Next, the CPU 24 performs a process for detecting a change in the position of the subject 2 based on the color image and the distance image generated by the data processing unit 20, and further acquires position data after the movement of the subject 2.

  That is, when the data processing unit 20 generates a color image, the CPU 24 acquires a new color image from the data processing unit 20 and acquires the already acquired color image and distance image from the nonvolatile memory 23.

  When the data processing unit 20 generates a distance image, the CPU 24 acquires a new distance image from the data processing unit 20 and acquires the already acquired distance image from the nonvolatile memory 23.

  Then, the CPU 24 compares the two color images and selects an analysis method for detecting a change in the position of the subject 2. Further, the CPU 24 compares two distance images and selects an analysis method for detecting a change in the position of the subject 2.

  The following three methods can be considered as analysis methods. First, analysis method 1 is a method of detecting the position of subject 2 by a pattern matching method.

  The analysis method 2 is a method of detecting the position of the subject 2 by dividing the imaging area 3 into small areas and detecting a change in entropy of each divided small area. Entropy is the occurrence probability of an event at a certain pixel in the screen.

  The analysis method 3 is a method of detecting the position of the subject 2 by dividing the imaging area 3 into small areas and detecting changes in the color difference signals of the divided small areas.

  In the case of a color image, the CPU 24 determines the movement of the subject 2 using any one of analysis methods 1 to 3.

  In the case of a distance image, since the color difference component is not included in the distance image, the CPU 24 cannot use the analysis method 3 and analyzes the position of the subject 2 using the analysis method 1 or 2.

  And in the case of a color image, CPU24 selects from the said analysis methods 1-3, compares two color images, and detects the change of the position of the to-be-photographed object 2 according to the selected analysis method.

  In the case of a distance image, the CPU 24 selects the analysis method 1 or 2, compares the two distance images, and detects a change in the position of the subject 2 according to the selected analysis method.

  The CPU 24 compares the two color images to determine whether or not the position of the subject 2 has changed. If the CPU 24 determines that the position has changed, the CPU 24 acquires the position after the movement of the subject 2.

  Then, the CPU 24 stores the newly acquired color image data and the position data after the movement of the subject 2 in the RAM.

  Similarly for the distance image, the CPU 24 compares the two distance images to determine whether or not the position of the subject 2 has changed, and if it determines that the position has changed, the CPU 24 determines the position after the movement of the subject 2. get.

  Then, the CPU 24 stores the newly acquired distance image data and the position data after the movement of the subject 2 in the RAM.

  The CPU 24 determines whether or not the two position data stored in the RAM are valid. If the CPU 24 determines that the two position data are valid, the CPU 24 stores the new position data stored in the RAM in the nonvolatile memory 23. To do.

  That is, it is determined whether or not the two position data are valid based on how the subject 2 has moved and the movement history.

  The possibility that the subject 2 appears at a completely different position from the past is low, and the possibility that the subject 2 suddenly changes its traveling direction is also low.

  Therefore, the CPU 24 sets a range in the moving direction of the subject 2 based on the position data stored in the nonvolatile memory 23 and determines that the position data within this range is valid.

  The CPU 24 stores the position data determined as valid in this manner in the nonvolatile memory 23. Note that the CPU 24 sets a limit number in advance for the number of data stored in the nonvolatile memory 23, and deletes old data when the number of stored data exceeds this limit number.

  Next, the CPU 24 selects one of the acquired position data as a result of comparing the two color images and the acquired position data as a result of comparing the two distance images.

  If the acquired position data overlaps with the acquired position data as a result of comparing the two color images, and the acquired position data as a result of comparing the two distance images, the CPU 24 moves the overlapping position data of the subject 2. Adopt as the data of the previous position.

  When the CPU 24 acquires the position data after the movement of the subject 2 as described above, the CPU 24 moves the tracking area FA to the position after the movement of the subject 2, supplies the control signal Sc1 to the lens driving unit 17, and supplies the lens driving unit 17 to the lens driving unit 17. And the subject 2 is focused in the new tracking area FA.

Next, the operation of the imaging apparatus 1 according to the first embodiment will be described.
For example, when the imaging apparatus 1 is provided in a digital camera, when the half shutter is pressed, the CPU 24 reads program data for AF processing from the ROM 21. The CPU 24 repeatedly executes this AF process at a preset cycle while the half shutter is pressed according to the flowcharts shown in FIGS.

  The CPU 24 supplies the timing signal St to the flash control unit 14, the imaging control unit 15, and the sensor unit 11 (step S11).

  The flash control unit 14 emits the flash 12 and projects visible light toward the subject 2, and the imaging control unit 15 emits the flash 13 and directs the pulse-modulated near infrared light toward the subject 2. Project.

  When the timing signal St is supplied from the CPU 24, the sensor unit 11 generates color image data and distance image data, and the memory controller 19 converts the color image data and distance image data into color image areas of the frame memory 18, respectively. 18a and the distance image area 18b.

  The data processing unit 20 generates a color image and a distance image based on the data stored in the color image area 18a and the distance image area 18b of the frame memory 18.

  The CPU 24 acquires a new color image generated by the data processing unit 20 (step S12).

  The CPU 24 acquires a color image from the nonvolatile memory 23 (step S13).

The CPU 24 compares the newly acquired color image with the color image acquired from the nonvolatile memory 23 (step S14).
The CPU 24 selects an analysis method based on the comparison result (step S15).

  The CPU 24 analyzes the change in the position of the subject 2 using the selected analysis method (step S16).

  The CPU 24 determines whether or not the position of the subject 2 has changed as a result of the analysis (step S17).

  When it is determined that the position of the subject 2 has not changed (No in step S17), the CPU 24 executes the process of step S20.

  On the other hand, when it is determined that the position of the subject 2 has changed (Yes in step S17), the CPU 24 acquires the position after the movement of the subject 2 (step S18).

  The CPU 24 stores the newly acquired color image and the position of the subject 2 after movement in the RAM (step S19).

  The CPU 24 acquires a new distance image generated by the data processing unit 20 (step S20).

  The CPU 24 acquires a distance image from the nonvolatile memory 23 (step S21).

The CPU 24 compares the newly acquired color image with the color image acquired from the nonvolatile memory 23 (step S22).
The CPU 24 selects an analysis method based on the comparison result (step S23).

  The CPU 24 analyzes the change in the position of the subject 2 using the selected analysis method (step S24).

  The CPU 24 determines whether or not the position of the subject 2 has changed as a result of the analysis (step S25).

  If it is determined that the position of the subject 2 has not changed (No in step S25), the CPU 24 ends the AF process.

  On the other hand, when it is determined that the position of the subject 2 has changed (Yes in step S25), the CPU 24 acquires the position after the movement of the subject 2 (step S26).

  The CPU 24 stores the newly acquired distance image and the position of the subject 2 after movement in the RAM (step S27).

  The CPU 24 sets the moving direction of the subject 2 based on the position data that the subject 2 has moved so far stored in the nonvolatile memory 23 (step S28).

  The CPU 24 determines whether or not the moved position of the subject 2 stored in the RAM is within the set moving direction range (step S29).

  If it is determined that the moved position of the subject 2 stored in the RAM is not within the set moving direction range (No in step S29), the CPU 24 ends the AF process.

  On the other hand, when it is determined that the position after movement of the subject 2 stored in the RAM is within the set moving direction range (Yes in step S29), the CPU 24 is based on the color image data and color image stored in the RAM. The position data, the distance image, and the position data acquired based on the distance image are stored in the nonvolatile memory 23 (step S30).

  The CPU 24 selects one of position data acquired based on the color image and position data acquired based on the distance image (step S31).

  The CPU 24 moves the tracking area FA based on the selected position data (step S32).

  The CPU 24 supplies the control signal Sc1 to the lens driving unit 17 and controls the lens driving unit 17 so as to focus on the subject 2 (step S33).

The CPU 24 determines whether or not the subject 2 is in focus (step S34).
If it is determined that the subject is not in focus (No in step S34), the CPU 24 supplies the control signal Sc1 to the lens driver 17 again to control the lens driver 17 so that the subject 2 is in focus. (Step S33).

  If it is determined that the subject is in focus (Yes in step S34), the CPU 24 ends the AF process.

Next, the operation of the imaging apparatus 1 will be specifically described.
The CPU 24 supplies the timing signal St to the flash control unit 14, the imaging control unit 15, and the sensor unit 11 (processing in step S11).

  The color image 2Di0 shown in FIG. 13A is a color image stored in the nonvolatile memory 23, and the color image 2Di1 shown in FIG. 13B is a color image newly acquired by the data processing unit 20.

  The CPU 24 acquires the color image 2Di1 shown in FIG. 13B from the data processing unit 20 (processing in step S12). Further, the CPU 24 acquires the color image 2Di0 shown in FIG. 13A from the nonvolatile memory 23 (processing in step S13).

  The CPU 24 compares the two color images 2Di0 and 2Di1 and selects an analysis method (steps S14 and S15).

  In this case, any of analysis methods 1 to 3 can be selected. For example, when the analysis method 2 is selected, the CPU 24 analyzes the change in the position of the subject 2 according to the analysis method 2 (processing in step S16).

  First, the CPU 24 sets areas UL, UR, LL, LR, and FA in the imaging area 3, as shown in FIGS. 13 (a) and 13 (b).

  Then, the CPU 24 calculates the entropy Pi of each region UL, UR, LL, LR, FA shown in FIGS. 13A and 13B and analyzes the change in the position of the subject 2.

Assuming that entropy is Hi and the occurrence probability of event “i” in a pixel in each region is Pi, entropy Hi is expressed by the following equation (6).
Hi = -PilogPi (6)

  The CPU 24 calculates the entropy Hi of each region UL, UR, LL, LR, FA shown in FIGS. 13A and 13B according to the equation (6).

  As shown in FIGS. 13A and 13B, the entropy Hi of the areas FA and UR changes. For this reason, the CPU 24 determines that the position of the subject 2 has changed (Yes in step S17).

  As shown in FIG. 13C, the CPU 24 acquires the motion vector Vi1, and acquires the position after the movement of the subject 2 (processing in step S18).

  The CPU 24 stores the newly acquired color image 2Di1 and the motion vector Vi1 in the RAM (step S19).

The CPU 24 performs the same process on the distance image.
The distance image 2Dd0 illustrated in FIG. 14A is a distance image stored in the nonvolatile memory 23, and the distance image 2Dd1 illustrated in FIG. 14B is a distance image newly acquired by the data processing unit 20.

  The CPU 24 acquires the distance image 2Dd1 shown in FIG. 14B from the data processing unit 20 (the process of step S20), and acquires the distance image 2Dd0 shown in FIG. 14A from the nonvolatile memory 23 (step S21). Processing).

  The CPU 24 compares the two distance images 2Dd0 and 2Dd1 and selects an analysis method (processing in steps S22 and S23).

  Also in this case, since any of the analysis methods 1 to 3 can be selected, the CPU 24 selects the analysis method 2. The CPU 24 analyzes the change in the position of the subject 2 according to the analysis method 2 (processing in step S24).

  Similarly, as shown in FIGS. 14A and 14B, the CPU 24 sets areas UL, UR, LL, LR, and FA in the imaging area 3, as shown in FIGS. 14A and 14B. The entropy Pi of each region UL, UR, LL, LR, FA is calculated, and the change in the position of the subject 2 is analyzed.

  As shown in FIGS. 14A and 14B, since the entropy Hi of the areas FA and UR changes, the CPU 24 determines that the position of the subject 2 has changed (Yes in step S25).

  As shown in FIG. 14C, the CPU 24 acquires the motion vector Vd1 and acquires the position after movement of the subject 2 (processing in step S26).

  The CPU 24 stores the newly acquired color image 2Dd1 and the motion vector Vd1 in the RAM (step S27).

  As shown in FIG. 15A, when it is determined that the position after movement of the subject 2 obtained by the motion vectors Vi1 and Vd1 is within the set movement direction range (Yes in step S29), the CPU 24 stores data in the RAM. The stored data is stored in the nonvolatile memory 23 (step S30).

  In this case, since the post-movement position of the subject 2 obtained by the motion vectors Vi1 and Vd1 is within the set moving direction range, for example, the CPU 24 moves the subject 2 after the movement obtained by the motion vector Vi1. Is selected (processing in step S31).

  As a method for selecting position data, a motion vector that is estimated to have high detection accuracy is employed. Specifically, the CPU 24 stores the motion vector and estimates the detection accuracy based on the motion vector acquired so far.

  For example, the possibility that the color image and the distance image appear at completely different positions is low, and the possibility that the subject 2 suddenly changes the traveling direction is also low. In such a case, the CPU 24 determines that the detection accuracy is low and does not employ this position data.

  Then, as shown in FIG. 15B, the CPU 24 moves the tracking area FA based on the selected position data (step S32).

  The CPU 24 supplies the control signal Sc1 to the lens driving unit 17 (the process of step S33), and when in focus (Yes in step S34), the AF process is terminated.

As described above, according to the first embodiment, the CPU 24 compares the position data of the subject 2 acquired based on the color image with the position data acquired based on the distance image, and based on the comparison result. The change in the position of the subject 2 is acquired.
Therefore, the subject 2 can be focused with high accuracy, and the image of the subject 2 can be formed on the image sensor 101.

  In addition, the analysis methods 1 to 3 can be selected for the color image, and the analysis method 1 or 2 is selected for the distance image, and the position of the subject 2 changes. However, the subject 2 can be reliably tracked according to the situation.

  For example, as shown in FIG. 18, when the shape of the subject 2 changes as shown in (a) → (b) → (c) → (d), the CPU 24 stores the distance stored in the nonvolatile memory 23. The analysis method 2 is selected by comparing the image with the distance image generated by the data processing unit. When the CPU 24 selects the analysis method 2 and analyzes the change in the position of the subject 2, the change in the position of the subject 2 can be detected with high accuracy. Then, the image of the subject 2 can be formed on the image sensor with high accuracy.

  As shown in FIG. 19, when the object 2q having the same entropy as the subject 2p crosses in front of the subject 2p, the CPU 24 uses the color image stored in the nonvolatile memory 23 and the color generated by the data processing unit. The analysis method 3 is selected by comparing with the image.

  Then, the CPU 24 detects a change in the position of the subject 2 based on a change in the color difference signal of each small area.

  Thus, even if the object 2q having the same entropy as the subject 2p crosses in front of the subject 2p, the CPU 24 can track the subject 2 with high accuracy by selecting the analysis method 3, and with high accuracy. An image of the subject 2 can be formed on the image sensor.

  As shown in FIG. 20, when the subject 2 moves away from the imaging device 1, the CPU 24 compares the distance image stored in the nonvolatile memory 23 with the distance image generated by the data processing unit, and analyzes method 2. Select. When the CPU 24 selects the analysis method 2 and analyzes the change in the position of the subject 2, the change in the position of the subject 2 can be detected with high accuracy.

In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, when the subject 2 is out of the imaging area 3, the orientation of the imaging device 1 may be changed.

  In this case, as shown in FIG. 16, the imaging device 1 includes an imaging unit 1a, a control unit 1b, and an attitude control unit 1c. The imaging unit 1a is configured as shown in FIG. 17, and the ROM 21, RAM 22, nonvolatile memory 23, and CPU 24 shown in FIG. 1 are excluded from the imaging unit 1a. And the control part 1b is provided with ROM21, RAM22, the non-volatile memory 23, and CPU24.

  The posture control unit 1c is for controlling the posture of the imaging unit 1a to change the orientation.

  The CPU 24 of the control unit 1b outputs a timing signal to the imaging unit 1a, supplies a control signal Sc2 to the memory controller 19 of the imaging unit 1a, and controls the memory controller 19.

  Further, as shown in FIGS. 15 and 16, the CPU 24 acquires color image and distance image data Dout from the data processing unit 20 of the imaging unit 1 a.

  When the subject 2 is out of the imaging area 3, the CPU 24 supplies the control signal Sc3 to the posture control unit 1c and controls the posture control unit 1c so that the imaging unit 1a tracks the subject 2.

  By configuring the imaging device 1 in this way, even when the subject 2 is out of the imaging area 3, the control unit 1b and the posture control unit 1c adjust the orientation of the imaging unit 1a to track the subject 2. The subject 2 can be accurately captured in the tracking area FA.

  In this embodiment, both the color image and the distance image of the subject 2 are acquired. However, it is also possible to manually select either one of the images. In this case, the imaging device 1 includes an operation unit (not shown) for inputting instruction information.

Moreover, in the said embodiment, what has a structure as shown in FIG. 9 was used as a distance image sensor which measures distance. However, the distance image sensor is not limited to this. For example, three distance pixels arranged every other pixel may be configured as the n ⁺ layer 311c, the p ⁺ layer 311b, and the n ⁺ layer 311d in FIG. With this configuration, the degree of freedom in design is improved.

  In the above embodiment, the pixel for measuring the distance is described as having a completely different structure from the pixel for measuring the luminance of RGB, but this need not be the case. That is, the distance can be measured using a CMOS or CCD.

  A normal CMOS or CCD can receive near-infrared rays and ultraviolet rays, and the current imaging device has a time for the near-infrared rays and ultraviolet rays to be reflected back to the subject due to its operation speed. Although there is a problem with the measurement accuracy to measure, when measuring a relatively long distance using strong near infrared rays or ultraviolet rays, the distance can be obtained without using an image sensor with a special structure. Can be measured, and if the performance of the imaging device is improved in the future, the distance can be measured, and the pixel for measuring the distance may be configured in this way.

In the above embodiment, the AF is performed following the detected subject. However, this need not be the case.
For example, a specific mark may be superimposed and displayed following the detected subject, thereby making it easy to understand how the subject is moving.

  Further, as in Japanese Patent No. 3750499, it may be configured to perform framing following auto-framing so that a specific area is zoomed and cut out with the detected object as a center (automatic framing). Even in situations where it is difficult to keep framing together, appropriate framing can be continued.

  Further, not only AF tracking, tracking of a specific mark, and tracking of framing described above may be performed alone, but they may be used in combination as necessary.

  In the above-described embodiment, the program is described as being stored in advance in a memory or the like. However, a program for operating the imaging device 1 as all or part of the device or executing the above-described processing is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk). ), Stored on a computer-readable recording medium such as MO (Magneto Optical disk), etc., installed on another computer, operated as the above-mentioned means, or may execute the above-mentioned steps .

  Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is a block diagram which shows the structure of the imaging device which concerns on embodiment of this invention. It is a figure which shows the example which the imaging device shown in FIG. 1 images a to-be-photographed object. It is a figure which shows an image when an imaging device image | photographs a to-be-photographed object, (a) shows the color image of the to-be-photographed object which the imaging device acquired, (b) shows the distance image which the imaging device acquired. It is a figure which shows the structure of the image pick-up element shown in FIG. It is a figure which shows the structure of the color image acquisition part with which the imaging device shown in FIG. 1 is provided. 6 shows a configuration of a pixel circuit of the color image acquisition unit shown in FIG. It is a figure which shows the structure of the distance image acquisition part with which the imaging device shown in FIG. 1 is provided. It is a timing chart which shows operation | movement of the distance image acquisition part shown in FIG. 8A and 8B are diagrams illustrating a configuration of a pixel circuit of the distance image acquisition unit illustrated in FIG. 7, in which FIG. 7A illustrates a photodiode and a cross-section of the periphery of the pixel circuit, and FIG. 8B illustrates an equivalent circuit of FIG. . It is a figure which shows the tracking area | region set in the imaging area. FIG. 3 is a flowchart (No. 1) illustrating an AF process executed by a CPU illustrated in FIG. 1. FIG. FIG. 3 is a flowchart (No. 2) showing an AF process executed by a CPU shown in FIG. 1. FIG. It is a figure which shows the specific operation | movement (the 1) of an imaging device, (a) shows each area | region set around the color image and tracking area | region memorize | stored in the non-volatile memory, (b) The color image acquired from the data processing unit and each area set around the tracking area are shown, and (c) shows the motion vector of the moved subject. It is a figure which shows the specific operation | movement (the 2) of an imaging device, (a) shows each area | region set around the distance image and tracking area | region memorize | stored in the non-volatile memory, (b) The distance image acquired from the data processing unit and each area set around the tracking area are shown, and (c) shows the motion vector of the moved subject. It is a figure which shows the specific operation | movement (the 3) of an imaging device, (a) is a motion vector acquired as a result of comparing two color images as a result of comparing two color images. (B) shows the moved tracking area. As an application example of the imaging apparatus, it is a diagram illustrating a configuration of the imaging apparatus in a case where a subject outside the imaging area is tracked. It is a figure which shows the structure of the imaging part shown in FIG. It is a figure which shows the to-be-photographed object to which a shape changes. It is a figure which shows the example which the object which has the same entropy of a to-be-photographed object crosses in front of a to-be-photographed object. It is a figure which shows the example to which a to-be-photographed object moves away from an imaging device.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Subject, 11 ... Sensor part, 14 ... Flash control part, 15 ... Imaging control part, 20 ... Data processing part, 24 ... CPU, 23: Non-volatile memory, 102, 103: Element drive unit, 200: Color image acquisition unit, 300: Distance image acquisition unit

Claims (18)

An image sensor that repeatedly images a subject;
A first image acquisition unit for acquiring a first image representing the shape of the subject including visual information of the subject obtained visually from the image of the subject captured by the imaging device;
A second image acquisition unit that acquires a second image representing the shape of the subject including distance information to the subject from the image of the subject captured by the imaging device;
Each time the image sensor performs imaging, the position of the subject is changed based on the first image acquired by the first image acquisition unit or the second image acquired by the second image acquisition unit. A position change detection unit to detect;
A tracking region moving unit that tracks the subject and moves a tracking region for performing specific control when the position change detection unit detects a change in the position of the subject.
An imaging apparatus characterized by that. The tracking area moving unit performs autofocus control as the specific control, and continues to acquire an image while performing autofocus control on the tracking area.
The imaging apparatus according to claim 1. The follow-up area moving unit performs superimposed display control of a predetermined image as the specific control, and continues to display the predetermined image superimposed on the follow-up area.
The imaging apparatus according to claim 1 or 2, wherein The tracking area moving unit performs auto framing control as the specific control, and keeps the frame following the tracking area.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. A storage unit that stores the first image acquired by the first image acquisition unit and the second image acquired by the second image acquisition unit each time the imaging element performs imaging;
The position change detection unit compares the first image stored in the storage unit with the first image newly acquired by the first image acquisition unit, and is stored in the storage unit. Detecting a change in the position of the subject based on one of the comparison results by comparing the second image and the second image newly acquired by the second image acquisition unit;
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The position change detector is configured to acquire the first object newly acquired by the subject and the first image acquisition unit included in the first image stored in the storage unit each time the image sensor performs imaging. A change in the position of the subject is detected in accordance with a first analysis method that compares the subject included in the two images and analyzes the positional relationship of the subject included in each of the two compared first images by pattern matching. To
The imaging apparatus according to claim 5. The position change detection unit compares the first image stored in the storage unit with the first image newly acquired by the first image acquisition unit every time the image sensor performs imaging. Then, a second analysis for analyzing the positional relationship of the subject included in the two first images compared by obtaining the entropy of each of the divided small areas by dividing the imaging area into a plurality of small areas. Detecting a change in the position of the subject according to a method;
The imaging apparatus according to claim 5. The position change detector is configured to acquire the first object newly acquired by the subject and the first image acquisition unit included in the first image stored in the storage unit each time the image sensor performs imaging. The subject included in the first image is included in the two first images compared by comparing the subject included in the image and obtaining the color difference signal of each small region divided by dividing the imaging region into a plurality of small regions. Detecting a change in the position of the subject according to a third analysis method for analyzing the positional relationship of the subject;
The imaging apparatus according to claim 5. The position change detection unit acquires the first image or the second image and the first image acquisition unit or the second image acquisition stored in the storage unit every time the image sensor performs imaging. And the first image stored in the storage unit and the first image when another object crossing the front and rear of the subject is detected by comparing the image with the newly acquired image by the unit The acquisition unit compares the subject included in the first image newly acquired, selects the third analysis method, and detects a change in the position of the subject according to the selected third analysis method. ,
The imaging apparatus according to claim 8. The position change detection unit compares the second image stored in the storage unit with the second image newly acquired by the second image acquisition unit every time the image sensor performs imaging. And detecting a change in the position of the subject according to a first analysis method for analyzing the positional relationship of the subject included in each of the two compared second images by pattern matching.
The image pickup apparatus according to claim 5, wherein the image pickup apparatus is an image pickup apparatus. The position change detection unit compares the second image stored in the storage unit with the second image newly acquired by the second image acquisition unit every time the image sensor performs imaging. Then, a second analysis for analyzing the positional relationship of the subject included in the two second images compared by acquiring the entropy of each of the divided small areas by dividing the imaging area into a plurality of small areas. Detecting a change in the position of the subject according to a method;
The image pickup apparatus according to claim 5, wherein the image pickup apparatus is an image pickup apparatus. The position change detection unit acquires the first image or the second image and the first image acquisition unit or the second image acquisition stored in the storage unit every time the image sensor performs imaging. The second image stored in the storage unit and the second image according to the second analysis method when a change in the shape of the subject is detected by comparing the newly acquired image with the image A change in the position of the subject is detected based on a comparison result with the second image newly acquired by the acquisition unit;
The imaging apparatus according to claim 11. Each time the position change detection unit captures an image,
Each time the first image acquisition unit and the second image acquisition unit acquire the first image and the second image, respectively, based on the position data of the plurality of subjects stored in the storage unit. Set the range of movement direction of the subject,
The position of the subject included in the first image newly acquired by the first image acquisition unit, and the position of the subject included in the second image newly acquired by the second image acquisition unit When the position is outside the set moving direction, the newly acquired position data of the subject is discarded, and only when the position is within the determined moving direction, the newly acquired position data of the subject is stored in the storage unit. Remember
The image pickup apparatus according to claim 5, wherein the image pickup apparatus is an image pickup apparatus. The image sensor receives light reflected by the subject, and includes a first pixel unit for acquiring the first image and a second pixel unit for acquiring the second image. As the pixel unit, a plurality of pixel units are arranged in a matrix.
A first element driving unit that drives the plurality of first pixel units so as to acquire the first image;
A second element driving unit that drives the plurality of second pixel units so as to acquire the second image.
The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. Each of the first pixel units includes a plurality of light receiving units that receive the light reflected by the subject for each color component to generate a signal charge, and output the signal charge generated for each color component. ,
The first element driving unit drives the plurality of light receiving units for each of the color components.
The imaging apparatus according to claim 14. The second image acquisition unit includes:
A light projecting unit that projects light;
An imaging control unit that blinks the light projected by the light projecting unit;
A plurality of second pixel portions of the image sensor;
A second element driving unit for driving the plurality of second pixel units;
A calculation unit that obtains distance information to the subject based on the signal charges output from each of the second pixel units,
Each of the second pixel units generates a signal charge by receiving the flashing light projected from the light projecting unit and reflected by the subject, and the generated signal charge is a period during which the light projecting unit is lit. The first signal charge generated by the reflected light received in the light and the second signal charge generated by the reflected light received during the period when the light projecting part is turned off are output separately.
Based on the amount of the first signal charge and the amount of the second signal charge output from each second pixel unit, the arithmetic unit turns on the light projecting unit and then turns on the light receiving unit. Measure the delay time until the reflected light is received, and obtain distance information to the subject based on the measured delay time.
The imaging apparatus according to claim 14 or 15, wherein Causing the image sensor to repeatedly image the subject;
Obtaining a first image representing the shape of the subject including visual information of the subject obtained visually from an image captured each time the image sensor performs imaging;
Obtaining a second image representing the shape of the subject, including distance information to the subject, from an image captured each time the image sensor performs imaging;
Detecting a change in the position of the subject based on the first image or the second image acquired each time the image sensor performs imaging;
Moving a follow-up area for performing specific control, which is set to track the subject when detecting a change in the position of the subject,
An object tracking method for an imaging apparatus. On the computer,
A procedure for repeatedly imaging a subject with an image sensor;
A procedure for acquiring a first image representing the shape of the subject including visual information of the subject obtained visually from an image captured each time the image sensor performs imaging,
A procedure for acquiring a second image representing the shape of the subject including distance information to the subject from an image captured each time the image sensor performs imaging,
A procedure for detecting a change in the position of the subject based on the first image or the second image acquired each time the subject is photographed;
A procedure for moving a tracking area for performing specific control, which is set to track the subject when detecting a change in the position of the subject;
A program for running

Images