JP4935380B2 - Image tracking device and imaging device - Google Patents

Description

  The present invention relates to an image tracking device and an imaging device.

  A frame (tracking frame) that surrounds the subject to be tracked is set in the captured image, and luminance information in the tracking frame is stored as a template (reference image). An image tracking device that searches for a high area and moves a tracking frame to the area to track a subject is known (for example, see Patent Document 1).

Prior art documents related to the invention of this application include the following.
Japanese Patent Laid-Open No. 07-0922248

  However, the conventional image tracking device has a problem that if a subject that is not a tracking target is mistakenly captured during template matching, it is difficult to return to the original tracking target subject.

The image tracking device according to claim 1 , an image sensor that captures an image formed by an optical system, and a reference image setting unit that sets a target portion of the image captured by the image sensor as a reference image. And calculating the degree of similarity between each of the plurality of partial areas in the image captured by the image sensor and the reference image, and selecting the positions of the plurality of partial areas selected in descending order of the degree of similarity among the plurality of partial areas. A candidate extraction unit that extracts a plurality of candidates for the new position of the target, a recognition unit that recognizes one of the plurality of candidates as a new position of the target, and a new image picked up by the imaging device that based on the position corresponding to the plurality of candidates of the following image, as well as a candidate for the next image by extracting a new plurality of candidates from the next image by the candidate extracting means, And a recognizing control means the new position of the target in the next image by said recognition means from the candidates for the next image, the control unit sets the candidate of the predetermined number of high order of similarity, If any of them is out of the candidates, a partial area having a high similarity with the candidate image having the highest similarity in the previous extraction by the candidate extraction unit is searched, and the partial area is newly set. It is set as the candidate.
The image tracking device according to claim 2, an image pickup device that picks up an image formed by an optical system, and a reference image setting unit that sets a target portion as a reference image among images picked up by the image pickup device. And calculating the degree of similarity between each of the plurality of partial areas in the image captured by the image sensor and the reference image, and selecting the positions of the plurality of partial areas selected in descending order of the degree of similarity among the plurality of partial areas. Corresponding to each of the plurality of candidates, candidate extracting means for extracting as a plurality of candidates for the new position of the target, recognition means for recognizing one of the plurality of candidates as the new position of the target, and Based on the position where the image is picked up, the candidate extraction unit newly extracts the plurality of candidates from the next image newly picked up by the imaging device, thereby updating the respective candidates. Together, characterized in that it comprises a recognizing control means the new position of the target in the next image by the recognition unit among the updated candidate.

  According to the present invention, even when another subject is mistakenly recognized as a target to be tracked, it can be returned to the original subject to be tracked.

  A single-lens reflex digital camera will be described as an example of an imaging device equipped with the image tracking device of the present invention. Note that the image pickup apparatus on which the image tracking device of the present invention is mounted is not limited to a single-lens reflex digital camera, but can be mounted on a compact digital camera or a single-lens reflex film camera.

  FIG. 1 is a block diagram showing an electric circuit of a single-lens reflex digital camera of one embodiment, and FIG. 2 is a cross-sectional view showing a mechanism of the single-lens reflex digital camera of one embodiment. In FIG. 1 and FIG. 2, illustration and description of camera devices and circuits not directly related to the present invention are omitted. The configuration of a single-lens reflex digital camera according to an embodiment will be described with reference to FIGS.

  The photometric element 1 is a sensor for detecting the subject brightness, and at the same time, is a sensor for analyzing the subject situation in order to track the subject, and the object scene has 22 lines as shown in FIG. The light is divided into 15 rows in a two-dimensional shape, and photometry is performed for each divided photometry area. The photometric element 1 is composed of a CCD, a CMOS, or the like, and a plurality of pixels are arranged in a two-dimensional form of 22 rows and 15 columns as shown in FIG. Further, each pixel is divided into pixels each provided with three primary color filters of B, G, and R as shown in FIG. The photometric circuit 2 performs signal processing such as noise removal, sensitivity correction, and A / D conversion on the photometric signal output from the photometric element 1, and outputs a BGR luminance signal for each pixel.

  The focus detection element 3 is built in the AF module 21 shown in FIG. 2, and passes through a pair of areas on the exit pupil plane of the photographing lens 22 in the focus detection areas set at 51 places in the object scene as shown in FIG. The shift amount of the pair of subject images formed by the pair of luminous fluxes is detected, and a focus detection signal for each focus detection area is output. FIG. 4 shows the correspondence between the divided photometry area of the photometry element 1 and the focus detection area of the focus detection element 3, and the 51 focus detection areas are set within the range of the 22 × 15 divided photometry area. Has been. The focus detection circuit 4 detects the defocus amount for each focus detection area by the phase difference detection method based on the focus detection signal for each focus detection area output from the focus detection element 3.

  The image pickup device 5 is composed of a CCD or a CMOS, picks up a subject image formed by the taking lens 22 and outputs an image pickup signal. The signal processing circuit 6 performs processing such as sampling, noise removal, sensitivity correction, and A / D conversion on the imaging signal output from the imaging device 5. The buffer 7 is a memory that temporarily stores image data, a defocus amount, and the like. The image processing circuit 8 performs processing such as pixel interpolation, resolution conversion, white balance control, gamma correction, matrix calculation, and image compression on the image data processed by the signal processing circuit 6. The memory card 9 is a recording medium that records captured images.

  The lens driving circuit 10 drives a focusing lens (not shown) of the photographic lens 22 according to the focus detection result, and adjusts the focus of the photographic lens 22. The aperture drive circuit 11 drives the aperture 23 according to the exposure calculation result to adjust the aperture. The shutter drive circuit 12 drives the shutter 24 according to the exposure calculation result to expose the image sensor 5. The mirror drive circuit 13 drives the main mirror 25 and the sub mirror 26 to raise and lower the mirrors 25 and 26. The display drive circuit 14 controls a diffusion type liquid crystal display (hereinafter referred to as a finder display) 15 provided on a screen 27 (see FIG. 2) of the finder optical system, and a tracking target region (referred to as a subject image on the screen 27). (Also called a tracking frame) and a focus detection area are superimposed and displayed.

  The operation switches 16 are switches that are turned on or off in accordance with the operation of various operation members for operating the camera. The operation switches 16 include a half-press switch that is turned on when the shutter button is half-pressed, a full-press switch that is turned on when the shutter button is fully pressed, a mode selector that selects a shooting mode such as a tracking shooting mode, and a focus detection area. The direction key for selecting is included.

  The control circuit 17 includes peripheral components such as a CPU 17a and a memory 17b, and is connected to the above-described camera circuits and devices via the bus 18 to perform various controls and operations of the camera. The control circuit 17 has a subject tracking function, a similarity calculation function, a distance calculation function, and the like depending on the software form of the microcomputer. Details of these functions will be described later.

  In FIG. 2, the main mirror 25 and the sub mirror 26 are placed in the optical path of the photographing lens 22 when not photographing. In this state, a part of the subject luminous flux transmitted through the photographing lens 22 is reflected by the main mirror 25 and guided to the diffusion screen 27 to form a subject image (hereinafter referred to as a finder image). This viewfinder image is guided to the photographer's eyes through a condenser lens 28, a pentaprism 29, a beam splitter 30, and an eyepiece lens 31, and a part of the viewfinder image is divided upward by the beam splitter 30 to form a diffraction grating 32 and a photometric lens 33. To the photometric element 1 via

  The diffraction grating 32 uses the light diffraction phenomenon to split the light flux for each wavelength. As shown in FIG. 5, the light emitted from one point of the subject by the diffraction grating 32 is split and incident on the photometric element 1, so that each of the RGB pixels of the photometric element 1 shown in FIG. Light enters. Thereby, false color correction can be performed, and correct color information can be detected when the subject is tracked based on the color information of the subject to be tracked.

  When not photographing, a part of the subject luminous flux is transmitted through the main mirror 25, reflected by the sub mirror 26 and guided to the AF module 21, and a pair of subject images for each focus detection area by the focus detection element 3 of the AF module 21. The amount of deviation is detected.

  At the time of photographing, the main mirror 25 and the sub mirror 26 are flipped up to the position 25 'and are retracted from the photographing optical path of the photographing lens 22. In this state, the aperture 23 is reduced to a predetermined aperture value, the shutter 24 is opened only for a predetermined shutter time, and imaging by the image sensor 5 is performed. After the shutter time elapses, the shutter 24 is closed, the main mirror 25 and the sub mirror 26 are returned to the photographing optical path, and the aperture 23 is opened.

  Next, the subject tracking method will be described. In this embodiment, the image data of the tracking target area (tracking frame) designated by the photographer or the tracking target area automatically set by the camera in the captured image is buffered as template image data (reference image data). 7 is searched for a region of image data that matches the template image data by a pattern matching method from the images that are repeatedly captured after that, and the region of the image data that matches the template image data is searched for the subject intended by the photographer. Considering the area, the tracking frame is moved to that area and the subject is tracked. In this embodiment, the RGB output of each pixel in the tracking target area (tracking frame) is stored as template image data.

  In the above pattern matching, the calculation area is defined so as to surround the tracking frame, and the RGB output of each pixel of the template image data and the comparison target area in the calculation area are gradually shifted in the calculation area in the vertical and horizontal directions. The RGB output of the image data is compared. Then, an output difference between corresponding pixels is obtained, and a sum of differences of all pixels is calculated as a residual amount. Residual amount calculation processing is performed while shifting the template up, down, left, and right within the calculation region, and it is assumed that the comparison target region with the smallest residual amount is the subject with the highest similarity, that is, the subject to be tracked. Note that the subject tracking method is not limited to the above-described method, and many already known methods can be used.

  In this embodiment, basically, the subject in the region where the residual amount is small by the subject tracking method described above, that is, the similarity is high is set as the tracking subject, but the tracking subject has the highest similarity. A plurality of additional fertilizer subject candidates are set in descending order of similarity. That is, the subject in the region with the highest similarity is the first candidate for the tracking subject, the subject in the region with the next highest similarity is the second candidate for the tracking subject, and the subject in the region with the next highest similarity is the tracking subject. Are set as third candidates. The tracking subjects from the first candidate to the third candidate are tracked simultaneously.

6 and 7 are diagrams for explaining a tracking method using a plurality of tracking subject candidates according to an embodiment. FIG. 6 shows the transition of the tracking subject candidate. FIG. 7 shows the tracking of the tracking subject candidate in the screen. Indicates the movement of the frame. The symbols Phase1, Phase2,... In the figure represent subject tracking scenes for each residual amount calculation. In FIG. 7 , a circular area represents a plurality of subjects in the screen, and a rectangular area represents a tracking subject candidate area. First, in Phase 1, the initial position of the tracking target area (tracking frame) is manually designated or automatically set on the image photographed by the photometric element 1. In Phase 1 of FIG. 7, a thick rectangular tracking frame is set for the subject indicated by the circular area in the center of the screen.

In Phase 2, the subject tracking process described above is performed on the next photographed image, and the first candidate 2-1, the second candidate 2-2, and the third candidate 2-3 of the tracking subject are calculated in descending order of similarity. In Phase 2 of FIG. 7 , the tracking frame of the first candidate is indicated by a bold rectangle, and the tracking frame of the second candidate and the third candidate is indicated by a thin rectangle. In the following description, every time the subject tracking process described above is performed, the image data in the tracking frame from the first candidate to the third candidate is stored in the buffer 7.

  In Phase 3, subject tracking processing is performed on the next captured image, and the first candidate 3-1, the second candidate 3-2, and the third candidate 3-3 of the tracking subject are calculated in descending order of similarity. Then, the difference in similarity between the candidates is obtained, and if the difference in similarity between the candidates is equal to or less than a predetermined threshold, the previous tracking frame is updated with the current tracking frame. If the difference between the similarity of any candidate and the similarity of another candidate is greater than a predetermined threshold, the candidate is excluded from the tracking subject candidates. In Phase 4, when subject tracking processing is performed on the next captured image and three tracking subjects are calculated in descending order of similarity, the similarity between the conventional first candidate and the second candidate is reversed, so the first candidate 4 -2, second candidate 4-1, and third candidate 4-3, and update each tracking frame.

In the subsequent Phase 5, when the subject tracking process for the next captured image is performed and three tracking subjects are calculated in descending order of similarity, the similarity detected in the tracking frame corresponding to the previous third candidate 4-3 is obtained. Since the threshold value is lower than the preset threshold value for determining the tracking subject (in this embodiment, the residual amount is higher than the preset tracking subject determination threshold value), the previous third candidate 4 The tracking frame corresponding to -3 is invalidated, and the subject surrounded by the tracking frame is not regarded as a tracking subject candidate. In Phase 5 of FIG. 7 , the tracking frame that is invalidated because the similarity is lower than the threshold value is indicated by a broken-line rectangle. Since the number of tracking subject candidates has decreased by one, the above-described subject tracking processing is performed using the image of the tracking frame adopted as the first candidate 4-2 in the previous Phase 4 as a template image, and the first candidate 5-2-1 and the second candidate 5-2-2 is calculated, and the tracking frame corresponding to the previous second candidate 4-1 is calculated as the third candidate 5-1. As a result, even if one tracking frame is invalidated, three tracking subject candidates can be set again.

In Phase 6, when the subject tracking process for the next captured image is performed and three tracking subjects are calculated in descending order of similarity, the similarity between the conventional first candidate and the second candidate is reversed. The tracking frame is updated as 2-2-1 and the second candidate 6-2-1. Furthermore, the tracking frame corresponding to the third candidate 5-1 of the previous Phase 5 is the distance on the screen to the tracking frame of the new first candidate 6-2-1 and the second candidate 6-2-1. Since it is larger than the threshold for determining a tracking subject, the tracking frame corresponding to the previous third candidate 5-1 is invalidated, and the subject surrounded by the tracking frame is not a tracking subject candidate. In Phase 6 of FIG. 7 , the tracking frame that has been invalidated due to a long distance is indicated by a broken-line rectangle. Since the number of tracking subject candidates has decreased by one, the above-described subject tracking processing is performed using the image of the previous tracking frame 5-2-2 of the current first candidate 6-2-2-1 as a template image, and the third candidate 6-2-1 Calculated as 2-2-2. As a result, even if one tracking frame is invalidated, three tracking subject candidates can be set again.

  In Phase 7, the subject tracking process described above is performed on the next captured image, and the first candidate 7-2-2-1, second candidate 7-2-1 and third candidate 7- of the tracking subject in descending order of similarity. 2-2-2 is calculated, and the difference in similarity between the candidates is obtained as described above. If the difference in similarity between the candidates is equal to or less than a predetermined threshold, the previous tracking frame is Update with the tracking frame.

  FIG. 8 is a flowchart illustrating subject tracking imaging processing according to an embodiment. The control circuit 17 executes this process when the tracking shooting mode is selected by the shooting mode selector included in the operation switches 16. In step 1, photometry of the object scene is performed by the photometric element 1 and the photometric circuit 2, and image data of the photometric result is stored in the buffer 7. Further, the focus detection element 3 and the focus detection circuit 4 detect the defocus amount for each focus detection area, and here, for example, the lens drive circuit 11 performs focus adjustment according to the defocus amount indicating the closest distance.

  In step 2, the focus detection area selected by the photographer is input with the direction key included in the operation switches 16 and the tracking target selected by the photographer is selected from the plurality of divided photometry areas of the photometry element 1 corresponding to the selected area. It is recognized as a region (tracking frame), and the tracking frame is superimposed on the finder image by the finder display 15 and displayed. In step 3, the release half-press switch of the operation switch 16 is confirmed, and if the release half-press operation is performed, the process proceeds to step 4; otherwise, the process returns to step 1 to repeat the above-described processing. When the release half-press operation is performed, the tracking frame selected by the photographer in step 4 is determined as the initial tracking frame.

  In this embodiment, an example in which the focus detection area is selected with the direction key and the initial tracking frame is set is shown. However, if a release half-press operation is performed while the subject to be tracked is captured in the center of the screen, the screen There are various methods for setting the initial tracking frame, such as the method for setting the initial tracking frame in the center and the method for automatically setting the tracking frame at the position of the focus detection area that indicates the closest defocus amount as a result of focus detection. The method can be adopted. In this embodiment, the size of the tracking frame is a 2-by-2 divided photometry area, but it may be a larger area.

  In step 5, photometry of the object scene is performed by the photometric element 1 and the photometric circuit 2, and image data of the photometric result is stored in the buffer 7. Further, the focus detection element 3 and the focus detection circuit 4 detect the defocus amount for each focus detection area, and store the defocus amount of each area of the detection result in the buffer 7. In the subsequent step 6, the above-described subject tracking processing is performed on the photometric result image (see FIGS. 6 to 7), the third candidate is calculated from the first candidate of the tracking subject, and the tracking frame of each candidate is displayed on the viewfinder display. 15 is displayed superimposed on the viewfinder image. In step 7, the defocus amount detected in the focus detection area corresponding to the tracking frame of the first candidate of the tracking subject is read from the buffer 7, and the lens drive circuit 10 adjusts the focus of the photographing lens 22 according to the defocus amount. Do.

  In step 8, the release full-push operation of the operation switches 16 is confirmed by the release full-push switch. If the release full-push operation is performed, the process proceeds to step 9; otherwise, the process returns to step 5 and the above-described processing is repeated. When the release full-press operation is performed, in step 9, the latest photometric result (photometric result in step 5) is read from the buffer 7, and the subject brightness is detected to perform exposure calculation. Then, the aperture is adjusted by the aperture driving circuit 11 with the aperture value obtained as a result of the exposure calculation, and the shutter 24 is driven by the shutter driving circuit 12 when the shutter time is the exposure calculation result. Next, in step 10, the mirror drive circuit 13 performs mirror up and the image sensor 5 captures an image. In step 11 after imaging, the image processing circuit 8 processes the captured image and records the image on the memory card 9.

  In the above-described embodiment, the description has been made on the assumption that the template image data of the initial tracking frame is not updated. However, the template image data may be updated for each subject tracking process described above. For example, the image data of the tracking frame of the first candidate of the tracking subject may be new template image data, or the average value or the weighted average value of the tracking data of the tracking frames of the first to third candidates is used as new template image data. It is good. Alternatively, the weighted average value of the template image data of the initial tracking frame and the image data of the first candidate tracking frame calculated for each subject tracking process may be used as new template image data.

  In the above-described embodiment, an example in which subject tracking is performed using an image captured by the photometric element 1 has been described. However, subject tracking is performed by using an image captured by the image sensor 5 at the time of non-shooting. Also good. Furthermore, in the above-described embodiment, an example in which three tracking subject candidates are tracked has been shown. However, the number of subject candidates to be tracked is not limited to three, and may be two or four or more. .

  According to one embodiment, a portion to be subject tracking is set as a reference image (template image) among images captured by the photometric element 1. Next, the degree of similarity between each of the plurality of partial areas in the image captured by the photometric element 1 and the reference image is obtained, and the positions of the plurality of partial areas selected in descending order of similarity among the plurality of partial areas are determined as targets. Are extracted as a plurality of candidates for the new position, and one of the candidates is recognized as a new position of the target. Then, based on positions corresponding to a plurality of candidates among the next images newly captured by the photometric element 1, a plurality of candidates are newly extracted from the next image to be candidates for the next image, and these The new position of the object in the next image is recognized from the candidates for the next image. As a result, even if the subject to be tracked is mistakenly recognized as a subject to be tracked by another subject, it can be returned to the original subject to be tracked. For example, when one of a plurality of soccer players wearing the same uniform is being tracked, even if another player is recognized as a tracking target depending on the player's orientation, it is the original tracking target at a later timing. It will return when the player's orientation changes.

  Further, according to one embodiment, when the similarity of the extracted candidate is lower than a predetermined value, the candidate is excluded from the candidate. Candidates can continue to be captured accurately.

  According to an embodiment, a difference in similarity between extracted candidates is calculated, and when a difference in similarity between any candidate and a plurality of other candidates is greater than a predetermined value, Since the candidates are excluded from the candidates, it is possible to continue to accurately capture a plurality of candidates that are likely to include the subject to be tracked.

  According to an embodiment, the distance between the extracted candidates on the image is calculated, and when the distance between any one of the candidates and the plurality of other candidates is greater than a predetermined value, the candidate is selected as a candidate. Therefore, it is possible to continue to accurately capture a plurality of candidates that are likely to include the subject to be tracked.

  According to one embodiment, a predetermined number of candidates are set in descending order of similarity, and if any of them is out of the candidates, the candidate image having the highest similarity in the previous extraction and Since a partial area having a high degree of similarity is searched and the partial area is set as a new candidate, a plurality of candidate areas that are likely to include the subject to be tracked can be accurately captured.

1 is a block diagram showing an electric circuit of a single-lens reflex digital camera of an embodiment; Cross-sectional view showing mechanism of single-lens reflex digital camera of one embodiment The figure which shows the structure of the photometric element of one embodiment. The figure which shows the correspondence of the division | segmentation photometry area of a photometry element, and the focus detection area of a focus detection element Diagram explaining the action of diffraction grating The figure for demonstrating the tracking method by the some tracking subject candidate of one Embodiment The figure for demonstrating the tracking method by the some tracking subject candidate of one Embodiment Flowchart showing subject tracking imaging processing of an embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Photometry element 4 Focus detection circuit 7 Buffer 9 Memory card 10 Lens drive circuit 16 Operation switch 17 Control circuit 22 Shooting lens

Claims (9)

An image sensor for imaging an image formed by an optical system;
Reference image setting means for setting a target portion as a reference image among images captured by the image sensor;
The similarity between the plurality of partial areas in the image captured by the image sensor and the reference image is obtained, and the positions of the plurality of partial areas selected in descending order of the similarity among the plurality of partial areas, Candidate extraction means for extracting as a plurality of candidates for the new position of the target;
Recognizing means for recognizing one of the plurality of candidates as a new position of the target;
Based on positions corresponding to the plurality of candidates in the next image newly picked up by the imaging device, the candidate extraction unit newly extracts the plurality of candidates from the next image, and And a control means for recognizing a new position of the target in the next image by the recognition means from among candidates for the next image,
The control means sets a predetermined number of the candidates in descending order of similarity, and when any of them is out of the candidates, the highest similarity is obtained in the previous extraction by the candidate extraction means. An image tracking apparatus that searches for a partial area having a high similarity to a candidate image and sets the partial area as a new candidate . An image sensor for imaging an image formed by an optical system;
Reference image setting means for setting a target portion as a reference image among images captured by the image sensor;
The similarity between the plurality of partial areas in the image captured by the image sensor and the reference image is obtained, and the positions of the plurality of partial areas selected in descending order of the similarity among the plurality of partial areas, Candidate extraction means for extracting as a plurality of candidates for the new position of the target;
Recognizing means for recognizing one of the plurality of candidates as a new position of the target;
Based on the position corresponding to each of the plurality of candidates, the candidate extraction unit newly extracts the plurality of candidates from the next image newly picked up by the imaging device, thereby updating the candidates. And an image tracking apparatus comprising: a control unit that recognizes a new position of the target in the next image by the recognition unit from the updated candidates . In the image tracking device according to claim 1 or 2,
The image tracking device according to claim 1, wherein when the similarity of the candidate extracted by the candidate extraction unit is lower than a predetermined value, the control unit removes the candidate from the candidate. In the image tracking device according to claim 1 or 2,
The control means calculates a difference in similarity between the candidates extracted by the candidate extraction means, and when a difference in similarity between any candidate and a plurality of other candidates is greater than a predetermined value. Is an image tracking device characterized in that the candidate is excluded from the candidates. In the image tracking device according to claim 1 or 2,
The control means calculates a distance on the image of the candidates extracted by the candidate extraction means, and if the distance between any candidate and a plurality of other candidates is larger than a predetermined value, the control means An image tracking apparatus, wherein a candidate is excluded from the candidates. In the image tracking device according to any one of claims 1 to 5 ,
The image tracking device, wherein the reference image setting unit sets the updated reference image based on an image at a position recognized as a new position of the target by the recognition unit . An imaging apparatus comprising the image tracking device according to any one of claims 1 to 6,
An image pickup apparatus comprising recording means for recording an image picked up by the image pickup element. The imaging apparatus according to claim 7,
A focus detection means for detecting a focus adjustment state of the optical system in a plurality of focus detection areas in a photographing screen of the optical system;
An imaging apparatus comprising: a focus adjustment unit that performs focus adjustment of the optical system based on a focus adjustment state detected in the focus detection region corresponding to the position recognized by the recognition unit. The imaging device according to claim 8,
A selection member for a photographer to select one of the plurality of focus detection areas in the imaging screen of the optical system;
The imaging apparatus according to claim 1, wherein the reference image setting unit targets an area corresponding to the focus detection area selected by the selection member in an image captured by the imaging element.

Images