JP2008046354A - Object tracking device and camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracking device capable of improving tracking accuracy by distinguishing an object as a tracking object from background information. <P>SOLUTION: The object tracking device is provided with: an imaging means 19 which images an image of an object formed by an imaging optical system 21 and generates picture information; a focus detection means 35 which detects a focus control state of the imaging optical system 21 for the object; and a tracking means 33 which sets reference information for tracking a target of an object image based on the picture information and focus control state and tracks the target. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a subject tracking device and a camera that automatically track a subject moving within a screen.

Background art and problems

There is known a camera that automatically tracks a subject when shooting a moving subject and performs focus detection and exposure calculation. In a conventional camera, an image serving as a reference for tracking a subject is stored, and a highly correlated object is tracked by pattern matching (for example, Patent Document 1). However, when similar image information is in the vicinity, the tracking accuracy cannot be ensured due to the influence.
JP 2006-58431 A

An object tracking device according to a first aspect of the present invention includes an imaging unit that captures an image of a subject by an imaging optical system to generate image information, a focus detection unit that detects a focus adjustment state of the imaging optical system with respect to the subject, Reference information for tracking a target in the subject image is set based on the image information and the focus adjustment state, and tracking means for tracking the target is provided.
According to a second aspect of the present invention, in the subject tracking device according to the first aspect, the reference information is a range of an image including a target in the image information.
According to a third aspect of the present invention, in the subject tracking device according to the first or second aspect, the tracking unit updates the reference information in accordance with a change in the focus adjustment state.
According to a fourth aspect of the present invention, in the subject tracking device according to the third aspect, the object tracking device further includes a determination unit that determines whether the target approaches or moves away from the camera based on the focus adjustment state. The reference information is changed based on the determination result of the means.
According to a fifth aspect of the present invention, in the subject tracking device according to any one of the first to fourth aspects, the recording and tracking unit performs tracking using at least one of color information and luminance in the image information. It is characterized by that.
According to a sixth aspect of the present invention, in the subject tracking device according to any one of the first to fifth aspects, the tracking unit selects a target candidate from the image information based on the hue value included in the reference information. It is characterized by that.
According to a seventh aspect of the present invention, in the subject tracking device according to the sixth aspect, the tracking means determines a target from the selected target candidates based on the focus adjustment state.
According to an eighth aspect of the present invention, a camera includes the subject tracking apparatus according to any one of the first to seventh aspects.

  According to the present invention, since the reference image is set based on the image information and the focus adjustment state and the target is tracked, the tracking accuracy can be improved.

  Hereinafter, an embodiment of a single-lens reflex digital camera including a subject tracking device will be described with reference to the drawings. FIG. 1 shows a main configuration of a digital camera of the present invention. A lens barrel 20 including a photographic lens 21 is attached to the camera body 10 in a replaceable manner.

  Inside the camera body 10, a first image sensor 12 for imaging a subject is provided. A CCD, CMOS, or the like is used as the first image sensor 12. A quick return mirror 11 that reflects subject light that has passed through the photographing lens 21 to the finder optical system is disposed between the photographing lens 21 and the first image sensor 12. Part of the subject light passes through the semi-transmissive region of the quick return mirror 11, is reflected downward by the sub mirror 111, and enters the phase difference AF sensor module 112.

  For example, the phase difference AF sensor module 112 divides a focus detection light beam into a pair of focus detection light images and a focus detection optical system that forms an image and a pair of split light images incident thereon and a focus corresponding thereto. And a pair of CCD line sensors that output detection signals. The focus detection signal output from the CCD line sensor is input to the control unit 30, and as will be described later, the control unit 30 outputs a lens drive signal for driving the focus adjustment lens to the in-focus position.

  The subject light reflected by the quick return mirror 11 forms an image on a focusing screen 13 provided at a position optically equivalent to the first image sensor 12. The subject image formed on the focusing screen 13 is observed by the photographer through the pentaprism 14 and the eyepiece lens 15 and passes through the prism 17 and the imaging lens 18 from the pentaprism 14. An image is formed on the light receiving surface of the image sensor 19. When a shooting operation is executed by fully pressing a release button (not shown), the quick return mirror 11 is rotated to a position indicated by a broken line in the drawing, and the subject light forms an image on the first image sensor 12. .

  As shown in FIG. 2A, the second image sensor 19 includes a plurality (20 × 14 in the illustrated example) of pixels 191 arranged in a matrix. Each pixel 191 is further divided into three fine pixels 191a to 191c as shown in FIG. Each of these fine pixels 191a to 191c is provided with three primary color filters of R (red), G (green), and B (blue), thereby obtaining an RGB output of the subject image.

  The second image sensor 19 causes the CCD control unit 31 to accumulate signal charges corresponding to the subject image formed on the imaging surface and sweep out the accumulated charges. The A / D conversion circuit 32 converts the analog image signal output from the second image sensor 19 into a digital image signal and outputs the digital image signal to the control unit 30. The control unit 30 includes a CPU, ROM, RAM, and various peripheral circuits. Functionally, the tracking control unit 33 that tracks the tracking target based on the image data acquired by the second image sensor 19, the focus detection calculation unit 35 that performs focus detection calculation, and the driving amount of the photographing lens 21. A lens drive amount calculation unit 36, and an on / off control unit 37 of a display unit 371 for turning on / off (turning on / off) an AF area mark representing a focus detection area 40 described later. The display unit 371 can display the AF area mark by the liquid crystal display plate 22 disposed in the vicinity of the focusing screen 13.

  The focus detection calculation unit 35 is based on a focus detection signal output from a pair of CCD line sensors corresponding to the focus detection region 40 selected by the photographer's operation of the area selection operation unit 351. Calculate the adjustment state.

  As shown in FIG. 3A, in the camera of this embodiment, for example, eleven focus detection areas 40a to 40k are set in a screen 41 such as a viewfinder field. When one of the eleven focus detection regions 40a to 40k, for example, the focus detection region 40e in FIG. 3A is selected by the area selection operation unit 351, the focus detection calculation unit 35 selects the selected focus detection region 40e. Is calculated for the focus adjustment state (hereinafter referred to as defocus amount for convenience). Furthermore, the area selection operation unit 351 outputs a selection region signal indicating which focus detection region is selected from among the eleven focus detection regions 40a to 40k to the tracking control unit 33. The defocus amount calculated by the focus detection calculation unit 35 is output to the lens drive amount calculation unit 36. The lens drive amount calculation unit 36 calculates a lens drive amount based on the input defocus amount and sends a lens drive signal to the lens drive motor 361. The lens drive motor 361 performs focus adjustment by moving the photographic lens 21 in the optical axis direction according to the lens drive signal. Further, the focus detection calculation unit 35 calculates the defocus amounts of all the focus detection areas 40a to 40k, and there is a difference from the defocus amount of the selected focus detection area 40e that is in the vicinity of the focus detection area 40e. A focus detection area 40 i within a predetermined value is detected, and a neighborhood area signal indicating the detected focus detection area is output to the tracking control unit 33.

  In the tracking control unit 33, based on the input selection region signal and the neighborhood region signal, as shown in FIG. 3B, the image data in a range including pixels corresponding to the focus detection regions 40e and 40i is defocused. Extracted as a quantity range 44. Further, on the initial image 43a acquired from the second image sensor 19, a group of pixels having a high correlation with the hue corresponding to the position of the focus detection area 40e is extracted as the same hue area 46 (hatched portion). A predetermined range including a range where the same defocus amount range 44 and the same hue region 46 overlap is set as a tracking subject region 47, and an image in the tracking subject region 47 is set as a template image 48 serving as a tracking reference. Further, a range enlarged by, for example, 3 pixels is set as the tracking search area 49 in the vertical and horizontal directions with the tracking subject area 47 as the center.

  Then, in the image acquired in time series by the second image sensor 19, a cutout area having the same size as the template image 48 is created within the tracking search area 49. The difference between the R (red), G (green), and B (blue) colors of the cutout area and the R (red), G (green), and B (blue) colors of the template image 48 is calculated. Perform subject tracking.

  The subject tracking method will be described in detail with reference to FIGS. Here, it is assumed that the photographer operates the area selection operation unit 351 to select the focus detection area 40e as shown in FIG. In response to this, the focus detection calculation unit 35 outputs the neighborhood region signal to the tracking control unit 33. The near region signal is a signal indicating the position of the focus detection region that is in the vicinity of the focus detection region 40e and has the same defocus amount as the focus detection region 40e. In the case of FIG. 3A, the region indicated by the neighborhood region signal is the focus detection region 40i.

  The tracking control unit 33 sets the image area corresponding to the focus detection areas 40e and 40i as the defocus amount range 44 based on the selection area signal and the vicinity area signal, as shown in FIG. . Furthermore, the hues in the image area corresponding to the focus detection area 40e, that is, R / G and B / G are set as subject hue values and stored in the storage unit 38. Thereafter, the same hue region 46 having substantially the same hue as the subject hue value in the entire captured image is detected. The tracking control unit 33 sets a predetermined range including, for example, 3 × 3 pixels including the overlapping portion of the same defocus amount range 44 and the same hue region 46 as the tracking subject region 47 as shown in FIG. To do. The image in the tracking subject area 47 is stored in the storage unit 38 as a template image 48a. Furthermore, a subject search area 49 is newly set on the upper, lower, left, and right outside the tracking subject area 47, for example, in a range enlarged by 3 pixels.

  FIG. 4A is a diagram showing the next image 43b acquired in time series from the second image sensor 19 following the initial image 43a. The tracking control unit 33 detects pixels within the range of the subject search area 49 that have hues within the predetermined range that are different from the subject hue values stored in the storage unit 38 (shaded portions in FIG. 4A). Then, a signal representing a focus detection area corresponding to the detected pixel position is output to the focus detection calculation unit 35. The focus detection calculation unit 35 calculates the defocus amount for the input focus detection region, that is, the focus detection regions 40a, 40d, and 40e in FIG. Based on the calculation result, the focus detection calculation unit 35 determines that the difference between the defocus amounts of the focus detection areas 40a, 40d, and 40e is equal to or less than a predetermined value, that is, the targets to be captured by the three focus detection areas are the same. It is determined whether or not. In FIG. 4A, the targets captured by the focus detection areas 40a and 40e are the same target. Further, the focus detection calculation unit 35 determines whether the tracking target is approaching or moving away from the camera body 10 in the optical axis direction based on the change in the defocus amount of the focus detection area 40e in the initial image 43a and the next image 43b. These determination results are output to the tracking control unit 33.

  The tracking control unit 33 determines whether or not there is a change in the size of the tracking target image on the image based on the input determination result. For example, when the approach to the tracking target camera is determined based on the defocus amounts in the focus detection areas 40a and 40e, the size of the template image to be tracked is enlarged. If it is determined that the tracking target is away from the camera, the size of the template image to be tracked is reduced. If it is not determined that the tracking target is approaching or moving away from the camera, the size of the template image to be tracked is not changed. When the tracking target is enlarged or reduced, the template image 48a stored in the storage unit 38 includes, for example, focus detection areas 40a and 40e having the same defocus amount by linear interpolation of the RGB values of each pixel. In addition, the template image 48b is created by enlarging or reducing so as not to overlap with other focus detection areas, and is stored in the storage unit 38.

  In the next image 43b shown in FIG. 4A, when the approach of the object is determined, the 3 × 3 pixel template image 48a is converted into a 4 × 4 pixel template image as shown in FIG. 4B. It is enlarged to 48b. If there is no change in the tracking target, the size of the template image 48a is not changed.

  After the template image is determined, the tracking control unit 33 sets the pixels having the same hue as the subject hue value (the hatched portion in FIG. 4A) as the tracking search pixel group 50. The tracking control unit 33 is arranged between the template image 48b and each 4 × 4 pixel cut-out area having the same size as the template image 48b, centering on each of the pixels indicated by the hatched portion in FIG. To calculate the difference between R, G, and B of each pixel. As a result of the calculation, the cut-out area with the smallest difference is set as a new tracking subject area 47b of 4 × 4 pixels, as shown in FIG. 4C.

  The tracking control unit 33 outputs the coordinates of the center position of the tracking subject area 47 b to the focus detection calculation unit 35. Based on the coordinates of the input tracking subject region 47b, the focus detection calculation unit 35, as shown in FIG. 4 (d), the focus detection region 40e closest to the center of the tracking subject region 47b captures the tracking target. The focus adjustment state is calculated using the focus detection signals output from the pair of CCD line sensors of the AF sensor module 112 corresponding to the focus detection area 40e. The focus detection calculation unit 35 outputs a selection region signal indicating the focus detection region 40 e to the lighting on / off control unit 37 via the tracking control unit 33. Thereby, the display unit 371 lights the AF area mark of the focus detection area 40e.

  When the plurality of focus detection areas 40 are located at the same distance from the center of the tracking subject area 47b, the movement direction of the subject in the time-series image acquired before that, that is, based on the movement history. Then, the focus detection area located in the direction in which the movement of the subject is predicted is selected.

  After the new tracking subject region 47b is set as described above, the tracking control unit 33 updates the template image 48. When the template image 48 is enlarged or reduced in accordance with the change in the size of the target image, the tracking control unit 33 tracks the color information of each color of R, G, and B in the template image 48b whose size has been changed and the tracking. The R, G, and B color information in the subject area 47b is combined. The tracking control unit 33 stores the updated template image in the storage unit 38. When the size of the template image 48 has not been changed, the tracking control unit 33 sets the color information of each color of R, G, B in the template image 48a and the color of each color of R, G, B in the new tracking subject region 47b. Combine with color information. Then, the tracking control unit 33 stores the updated template image 48 in the storage unit 38.

Hereinafter, each process of the subject tracking control operation will be described using the flowchart shown in FIG. Each process shown in this flowchart is performed by executing a program by the control unit 30. This flowchart is started by half-depressing a release switch (not shown).
In step S1, based on the focus detection calculation result calculated by the focus detection calculation unit 35 for the focus detection area set by the photographer, the photographing lens 21 is driven to adjust the focus, and the process proceeds to step S2. In step S2, an image from the second image sensor 19 is acquired, and the process proceeds to step S3.

  In step S3, the focus detection calculation unit 35 calculates the defocus amount for each of the focus detection areas 40a to 40k, and the process proceeds to step S4. In step S4, the same defocus amount range 44 is determined based on the defocus amount calculation result in step S3, and the process proceeds to step S5. In step S5, the hue of the image corresponding to the selected focus detection area 40 is set as the subject hue value, the subject hue value is stored in the storage unit 38, and the process proceeds to step S6.

  In step S6, the same hue area 46 having a hue substantially the same as the stored subject hue value is detected on the initial image, and the process proceeds to step S7. In step S7, a tracking subject region 47 is set based on the same defocus amount range 44 and the same hue region 46, and the process proceeds to step S8. In step S8, the image in the tracking subject area 47 set in step S7 is stored in the storage unit 38 as the template image 48, and the process proceeds to step S9.

  In step S9, the subject search area 49 is set as described above based on the tracking subject area 47 set in step S7, and the process proceeds to step S10. In step S10, it is determined whether or not a release switch (not shown) has been fully pressed by the photographer. If step S10 is positive, that is, if it is determined in step S10 that the release switch has been fully pressed, the process proceeds to step S16. If step S10 is negative, that is, if it is determined in step S10 that the release switch is not fully pressed, the process proceeds to step S11.

  In step S11, the next image (new image) is acquired from the second image sensor 19, and the process proceeds to step S12. In step S12, the update of the template image 48 and the pixel to be searched for the tracking target are determined by an update subroutine described later. After each process in the subroutine, the process proceeds to step S13. In step S13, the difference from the template image 48 (48a or 48b in the example of FIG. 4) is calculated for each of the cutout regions centered on each pixel of the subject search pixel group 50, and the process proceeds to step S14.

  In step S14, based on the calculation result in step S13, the cut-out area having the smallest difference value is determined as a new tracking subject area 47b, and the process proceeds to step S15. In step S15, the template image 48 is updated to a new template image 48 as described above, stored in the storage unit 38, and the process proceeds to step S16. In step S16, a selection area signal indicating the focus detection area closest to the center of the tracking subject area 47b specified in step S14 is output to the focus detection calculation unit 35, and the defocus amount for this focus detection area is calculated. The display unit 371 lights up the AF area mark corresponding to the focus detection area, and the process returns to step S10. If it is determined in step S10 that the release switch has been fully pressed, the process proceeds to step S17, where the photographing operation is performed and the series of processes is terminated.

The update subroutine of step S12 in FIG. 5 will be described using the flowchart shown in FIG.
In step S101, a pixel having a hue close to the hue value of the subject is detected in the subject search area 49, and the process proceeds to step S102. In step S102, a signal indicating the focus detection area corresponding to the pixel position detected in step S101 is output to the focus detection calculation unit 35, the defocus amount for this focus detection area is calculated, and the process proceeds to step S103.

  In step S103, based on the calculation result in step S102, even when there are a plurality of focus detection areas corresponding to pixels having a hue close to the hue value of the subject, whether or not these focus detection areas capture the same subject. And the process proceeds to step S104. In step S104, it is determined whether the tracking target is approaching or moving away from the optical axis direction of the camera body 10 based on the calculation result in step S102, and the process proceeds to step S105.

  In step S105, it is determined whether or not the template image 48 needs to be resized based on the determination results in steps S103 and S104. If it is necessary to change the size of the template image 48, an affirmative determination is made in step S105 and the process proceeds to step S106. If it is not necessary to change the size of the template image 48, a negative determination is made in step S105, and step S106 is skipped and the process proceeds to step S107.

  In step S106, the template image 48 is enlarged or reduced by the method described above, and the process proceeds to step S107. In step S107, pixels having a hue close to the subject hue value detected in step S101 are set as the tracking search pixel group 50. When the setting in step S107 is completed, the process exits the update subroutine and returns to the main routine.

According to the digital camera according to the embodiment described above, the following operational effects can be obtained.
(1) Reference information obtained from the range of the image including the tracking target based on the color information (image information) acquired by the second image sensor 19 and the focus adjustment state detected by the focus detection calculation unit 35. That is, the template image 48 is set to track the tracking target. Therefore, since the template image 48 can be set without the influence of background information different from the tracking target, tracking accuracy is improved.

(2) The template image 48 is updated according to the change in the focus adjustment state with respect to the tracking target. That is, when it is determined that the tracking target is approaching the camera body 10, the template image 48 is enlarged, and when it is determined that the tracking target is moving away, the template image 48 is reduced and the template image 48 is changed. Therefore, since the template image 48 is changed according to the size of the tracking target that changes on the screen depending on whether the tracking target approaches or moves away from the camera, the influence of the background information can be removed and the tracking accuracy can be improved. .

(3) The tracking subject region 47 is set on the image acquired by the second image sensor 19 based on the same hue region 46 and the same defocus amount range 44. The same hue area 46 is an area including a hue highly correlated with the hue in a predetermined range corresponding to the selected focus detection area, and the same defocus amount range 44 is substantially the same as the defocus amount of the selected focus detection area. This is a region including a focus detection region having a defocus amount of. That is, a subject to be a tracking target candidate is selected based on the hue value, and the tracking target is determined based on the focus adjustment state from the selected tracking target candidates. Therefore, since the tracking calculation can be performed without being influenced by background information different from the tracking target, the reliability of the tracking calculation can be maintained.

The embodiment described above can be modified as follows.
(1) Based on the hue of the tracking target acquired by the second image sensor 19, the subject hue value, the same hue region 46, and the like used as a reference when setting the template image 48 are determined. The brightness may be used.

(2) In the case of a camera having a function of recognizing a subject instead of specifying a subject to be tracked by a photographer's operation, the subject to be tracked is set based on the recognition result of the subject. The tracking process may be started.

(3) It has been described that the color information of each color of R, G, B of the template image 48 and the color information of each color of R, G, B of the tracking subject region 47b are combined and updated as a new template image 48. However, the tracking subject area 47b may be replaced as a new template image.

(4) Although described as a single-lens reflex digital camera with an interchangeable photographic lens, the present invention is not limited to this and may be applied to a lens-integrated camera or a video camera.

  In addition, the present invention is not limited to the above-described embodiment as long as the characteristics of the present invention are not impaired, and other forms conceivable within the scope of the technical idea of the present invention are also within the scope of the present invention. included.

It is a figure explaining the principal part structure of the digital camera in embodiment of this invention. It is a figure explaining the pixel composition of an image sensor. It is a figure which illustrates notionally various fields used for subject tracking control in an embodiment. It is a figure explaining the tracking method in embodiment. 6 is a flowchart illustrating an operation of subject tracking processing in the embodiment. It is a flowchart explaining the subroutine which determines the position and size of a tracking object in a subject tracking process.

Explanation of symbols

19 Second image sensor 33 Subject tracking control unit 35 Focus detection calculation unit 40 Focus detection area 44 Same defocus amount range 46 Same hue area 47 Tracking subject area 48 Template image 49 Subject search area 50 Subject search pixel group

Claims (8)

Imaging means for capturing an image of a subject by an imaging optical system and generating image information;
Focus detection means for detecting a focus adjustment state of the imaging optical system with respect to the subject;
A subject tracking apparatus comprising: tracking means for setting reference information for tracking a target in the subject image based on the image information and the focus adjustment state, and tracking the target. The subject tracking device according to claim 1,
The subject tracking device, wherein the reference information is a range of an image including the target in the image information. The subject tracking device according to claim 1 or 2,
The subject tracking device, wherein the tracking means updates the reference information according to a change in the focus adjustment state. The subject tracking device according to claim 3,
Determination means for determining whether the object approaches or moves away from the camera based on the focus adjustment state;
The subject tracking device, wherein the tracking unit changes the reference information based on a determination result of the determination unit. The subject tracking device according to any one of claims 1 to 4,
The subject tracking device, wherein the tracking means performs tracking using at least one of color information and luminance of the image information. In the subject tracking device according to any one of claims 1 to 5,
The subject tracking apparatus, wherein the tracking unit selects the target candidate from the image information based on a hue value included in the reference information. The subject tracking device according to claim 6,
The tracking apparatus according to claim 1, wherein the tracking unit determines the target based on the focus adjustment state from the selected target candidates.   A camera comprising the subject tracking device according to any one of claims 1 to 7.

Images