JP2013012944A - Tracker and tracking method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a tracker and a tracking method for performing appropriate tracking processing even in a case when tracking performance is deteriorated such as execution of special image processing, for example.SOLUTION: A tracker includes a system controller 130 for setting a first image generation mode for generating first image data being image data obtained by photographing or a second image generation mode for generating the first image data and generating second image data obtained by performing prescribed processing on the first image data, detecting feature information on the image data and detecting a face part of an object. The system controller 130 operates a moving position of a tracking object in response to the feature information or the face part detected from the image data. The system controller 130 switches a system to an operation different from an operation performed when the first image generation mode is set if the second image generation mode is set.

Description

  The present invention relates to a tracking device and a tracking method for tracking a subject.

  2. Description of the Related Art Conventionally, techniques for performing automatic focusing control (AF) or automatic exposure control (AE) so as to follow a specific subject when shooting a moving object or shooting a moving image are known. Tracking processing is used to follow such a specific subject. There are various types of tracking processes such as those using luminance information, those using color information, and those using wrinkle detection. Patent Document 1 discloses an automatic tracking device that performs a tracking operation on a subject image based on an imaging signal output from an imaging unit.

  In the automatic tracking device disclosed in Patent Document 1, parameters representing the characteristics of the subject (for example, the color of the subject and the background) are extracted for a preset tracking field of view, and the extracted features and newly extracted Based on the characteristics of the subject, the presence / absence of movement of the subject and, if the subject moves, the moving direction or moving position thereof are detected. Then, the tracking visual field is moved following the movement of the subject.

Japanese Patent No. 2605004

By the way, when only one type of tracking process is used, if the tracking process fails, the correct tracking process may not be possible thereafter. In view of such circumstances, a technique for performing a plurality of types of tracking processing has been proposed, but this has the following problems.
For example, when performing special image processing (for example, processing for generating a single image by combining a plurality of images), the load of the image processing increases. At this time, the tracking process including part of the image processing is naturally affected, and the tracking performance that is not generated in the normal tracking process may be deteriorated.

  Furthermore, for example, when performing tracking processing by combining a plurality of features, the input image data used for the tracking processing differs for each feature (for example, RAW data and YC data), and the generation frame rate of each input image data is set. If there is a difference, for example, for the “first feature”, the tracking process for the (n−1) th frame is completed, and for the “second feature”, the tracking process is completed only up to the n−4th frame. The phenomenon of not doing can happen.

  At this time, even if an attempt is made to synchronize the first feature and the second feature, there is a deviation from the current subject between the n-4 frames, and between the n-1 frame and the n-4 frame In the first place, since the frames are different, a correct tracking result cannot be obtained even if they are synchronized. That is, in any case, the tracking performance is degraded.

  The present invention has been made in view of the above-described circumstances. For example, a tracking device capable of performing appropriate tracking processing even when degradation of tracking performance, such as when performing special image processing, can occur. And to provide a tracking method.

In order to achieve the above object, the tracking device according to the first aspect of the present invention includes:
A first image generation mode for generating first image data, which is image data obtained by photographing, and a second for generating the first image data and performing predetermined processing on the first image data. A second image generation mode for generating the image data; a drawing mode setting unit for setting any one of the image generation modes;
An image processing unit that processes the image data input to the tracking device according to the setting of the drawing mode setting unit;
A tracking detection unit that detects feature information about the image data processed by the image processing unit and detects a face part of the subject;
A tracking calculation unit that calculates a movement position of the tracking target based on the feature information or the face part;
A calculation switching unit that switches the calculation by the tracking calculation unit to a calculation different from the calculation executed when the first image generation mode is set when the second image generation mode is set;
It is characterized by comprising.

In order to achieve the above object, the tracking method according to the second aspect of the present invention includes:
A first image generation mode for generating first image data, which is image data obtained by photographing, and a second for generating the first image data and performing predetermined processing on the first image data. One of the second image generation modes for generating the image data and the image generation mode, and
The input image data is processed according to the setting of the drawing mode setting unit,
Detecting feature information about the processed image data, and detecting a face part of the subject,
Based on the feature information or the face part, the movement position of the tracking target is calculated,
When the second image generation mode is set, the calculation by the tracking calculation unit is switched to a calculation different from the calculation executed when the first image generation mode is set.
It is characterized by that.

  According to the present invention, it is possible to provide a tracking device and a tracking method capable of performing appropriate tracking processing even when tracking performance may be deteriorated, for example, when performing special image processing. .

FIG. 1 is a diagram illustrating a configuration as an example of an imaging apparatus including a tracking device according to an embodiment of the present invention. FIG. 2 is a diagram for explaining tracking processing using luminance information. FIG. 3 is a diagram for explaining tracking processing using color information. FIG. 4 is a diagram for explaining wrinkle detection processing. FIG. 5 is a flowchart showing the shooting operation of the imaging apparatus. FIG. 6 is a flowchart showing a tracking process as a tracking method according to an embodiment of the present invention. FIG. 7 is a timing chart showing a tracking process as a tracking method according to an embodiment of the present invention. FIG. 8 is a timing chart showing a tracking process as a tracking method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration as an example of an imaging apparatus including a tracking device according to an embodiment of the present invention. An imaging apparatus 100 illustrated in FIG. 1 includes a photographing optical system 102, a focus adjustment mechanism 104, a diaphragm 106, a diaphragm driving mechanism 108, a shutter 110, a shutter driving mechanism 112, an imaging element 114, and an imaging element interface ( IF) circuit 116, RAM 118, display element 120, display element drive circuit 122, touch panel 124, touch panel drive circuit 126, recording medium 128, system controller 130, operation unit 132, and ROM 134. Have.

  The photographing optical system 102 is an optical system for condensing a light flux F from a subject (not shown) on the light receiving surface of the image sensor 114. The photographing optical system 102 has a plurality of lenses such as a focus lens. The focus adjustment mechanism 104 includes a motor and its drive circuit. The focus adjustment mechanism 104 drives the focus lens in the photographing optical system 102 in the optical axis direction (the one-dot chain line direction in the drawing) under the control of the CPU 1301 in the system controller 130.

  The diaphragm 106 is configured to be openable and closable, and adjusts the amount of the light beam F incident on the image sensor 114 via the photographing optical system 102. The aperture drive mechanism 108 has a drive mechanism for driving the aperture 106. The diaphragm driving mechanism 108 drives the diaphragm 106 according to the control of the CPU 1301 in the system controller 130.

  The shutter 110 is configured to place the light receiving surface of the image sensor 114 in a light shielding state or an exposure state. The exposure time of the image sensor 114 is adjusted by the shutter 110. The shutter drive mechanism 112 has a drive mechanism for driving the shutter 110, and drives the shutter 110 under the control of the CPU 1301 in the system controller 130.

  The imaging element 114 has a light receiving surface on which a light flux F from a subject condensed through the photographing optical system 102 is imaged. The light receiving surface of the image sensor 114 is configured by two-dimensionally arranging a plurality of pixels, and a color filter is provided on the light incident side of the light receiving surface. Such an image sensor 114 converts an image (subject image) corresponding to the light beam F formed on the light receiving surface into an electrical signal (hereinafter referred to as an image signal) corresponding to the light amount. Here, as the imaging device 114, imaging devices having various configurations such as a CCD system and a CMOS system are known. Various color filters such as a Bayer array are also known. In the present embodiment, the configuration of the image sensor 114 is not limited to a specific configuration, and image sensors having various configurations can be used. The imaging element 114 starts an exposure operation for imaging one frame at the rising timing (or falling timing may be) of the pulse of the vertical synchronization signal VD generated by the CPU 1301.

  The image sensor IF circuit 116 drives the image sensor 114 under the control of the CPU 1301 in the system controller 130. The image sensor IF circuit 116 reads out the image signal obtained by the image sensor 114 under the control of the CPU 1301 in the system controller 130, and performs CDS (correlated double sampling) processing or AGC (automatic operation) on the read image signal. Analog processing such as gain control processing is performed. Further, the image sensor IF circuit 116 converts the analog processed image signal into a digital signal, and generates image data (RAW data).

The RAM 118 is, for example, an SDRAM, and has a work area, an evaluation image area, a reference image area, a tracking position log area, and a tracking color area log area as storage areas.
The work area is a storage area provided in the RAM 118 for temporarily storing data generated in each unit of the imaging apparatus 100 such as image data (RAW data) obtained by the imaging element IF circuit 116.

  The evaluation image area is a storage area provided in the RAM 118 for temporarily storing evaluation image data. The evaluation image data is image data of a frame including a subject to be tracked in a tracking process described later. In the tracking process, processing is performed so as to track this tracking target.

  The reference image area is a storage area provided in the RAM 118 for temporarily storing reference image data. The reference image data is image data of a frame to be searched for a tracking target in tracking processing described later. In the tracking process, a search is performed in the reference image data.

  The tracking position log area is a storage area provided in the RAM 118 for temporarily storing the tracking position log. The tracking position log is a log in which the tracking position obtained as a result of the tracking process is recorded. In the present embodiment, a tracking target is tracked using a plurality of tracking processes in combination. Therefore, in the tracking position log, the tracking positions (candidate tracking positions) obtained by the respective tracking processes are individually recorded for the past 10 frames. In addition, the final tracking position adopted by the priority tracking position determination process described later is also recorded in the tracking position log.

  The tracking color area log area is a storage area provided in the RAM 118 for temporarily storing the tracking color area log. The tracking color area log is a log in which the tracking color area obtained by the tracking process is recorded. In this embodiment, the tracking color area is recorded for the past 10 frames. Details of the tracking color area will be described later.

  The display element 120 is a liquid crystal display (LCD), for example, and displays various images such as an image for live view and an image recorded on the recording medium 128. The display element driving circuit 122 drives the display element 120 based on the image data input from the CPU 1301 of the system controller 130 and causes the display element 120 to display an image.

  The touch panel 124 is integrally formed on the display screen of the display element 120, and detects a contact position of a user's finger or the like on the display screen. The touch panel drive circuit 126 drives the touch panel 124 and outputs a contact detection signal from the touch panel 124 to the CPU 1301 of the system controller 130. The CPU 1301 detects a contact operation on the display screen of the user from the contact detection signal, and executes processing according to the contact operation.

  The recording medium 128 is a memory card, for example, and records an image file obtained by a shooting operation. The image file is a file configured by adding a predetermined header to image data. In the header, data indicating shooting conditions, data indicating a tracking position, and the like are recorded as tag data.

  The system controller 130 is a control circuit for controlling the operation of the imaging apparatus 100. The CPU 1301, the AF control circuit 1302, the AE control circuit 1303, the image processing circuit 1304, the tracking processing circuits 1305 and 1306, and eyelid detection A circuit 1307, a color information acquisition circuit 1308, and a memory control circuit 1309 are included.

  The CPU 1301 includes blocks outside the system controller 130 such as the focus adjustment mechanism 104, the aperture driving mechanism 108, the shutter driving mechanism 112, the display element driving circuit 122, and the touch panel driving circuit 126, and each control circuit inside the system controller 130. Control the behavior. The CPU 1301 also has a function as a control unit, and determines a final tracking position in a priority tracking determination process described later. Further, the CPU 1301 generates a vertical synchronization signal VD for controlling the image sensor 114. Furthermore, the CPU 1301 causes the imaging apparatus 100 to execute “first image generation mode” in which “special image processing (image processing whose processing time is equal to or greater than a predetermined threshold)” by an image processing circuit 1304 described later, or “ The “second image generation mode” for executing “special image processing” is set. This setting is performed based on the operation of the operation unit 132 or the touch panel 124 by the user, for example. Each of these “image generation modes” settings means that an operation for causing the imaging apparatus to execute a corresponding “image generation process” is performed.

  The AF control circuit 1302 controls contrast AF processing. Specifically, the AF control circuit 1302 extracts high-frequency components of the image data (RAW data) obtained by the image sensor IF circuit 116, and integrates the extracted high-frequency components to thereby evaluate the focus for AF. Get the value. The CPU 1301 controls the focus adjustment mechanism 104 to bring the focus lens into focus while evaluating the contrast of the image data according to the focus evaluation value.

  The AE control circuit 1303 controls the AE operation. Specifically, the AE control circuit 1303 calculates the subject brightness using the image data (RAW data) obtained by the imaging element IF circuit 116. The CPU 1301 calculates the aperture amount (aperture value) of the aperture 106 during exposure, the opening time of the shutter 110 (shutter speed value), the image sensor sensitivity, the ISO sensitivity, and the like according to the subject brightness.

  The image processing circuit 1304 performs various image processing on the image data. As this image processing, compression processing, conversion processing from RAW data to YC data (image data composed of a luminance signal and a color difference signal), processing for combining a plurality of frames of image data to generate one frame of image data, This includes processing similar to optical processing or development performed on one frame of image data, processing of cutting out a part of one frame of image data, and generating a plurality of image data. Note that the image processing circuit 1304 also performs decompression processing on the compressed image data.

A tracking processing circuit 1305 as a first tracking position detection unit performs tracking processing using luminance information of image data as first feature information. The tracking process using this luminance information will be briefly described. In the tracking process using luminance information, for example, when a tracking target is set in the (N-1) frame shown in FIG. 2A, the image data of the (N-1) frame is used as the evaluation image data in the RAM 118. Is stored in the evaluation image area. Image data in a predetermined range 202 including the tracking target of the evaluation image data is set as reference image data. In the subsequent tracking process, a portion corresponding to the reference image data 202 of the reference image data is searched.
As an example of N frame tracking processing, N frame image data is first stored in the reference image area of the RAM 118 as reference image data. A portion of the reference image data corresponding to the reference image data 202 is searched for by obtaining a correlation amount between the image data in the predetermined search range 204 and the reference image data 202. The correlation amount is determined from, for example, the sum of absolute differences between the standard image data and the reference image data (obtained by obtaining absolute values of luminance differences for each pixel and integrating them). For example, if the sum of absolute differences between the reference image data in the reference image data area 206 and the standard image data 202 shown in FIG. 2B is obtained, the reference image data area 206 and the standard image data 202 are clearly shown. It is different image data, and the sum of absolute differences is increased. On the other hand, if the sum of absolute differences between the reference image data area 208 and the standard image data 202 is obtained, the sum of absolute differences becomes small. Thus, the difference absolute value sum decreases as the correlation amount with the reference image data 202 increases. In the tracking process using luminance information, an area having the maximum correlation amount, that is, the minimum difference absolute value sum is searched from the reference image data. In the example of FIG. In the tracking position log area, the position with the highest degree of matching in the area 208 is recorded as the tracking position. When there are a plurality of such positions, for example, a position close to the center of the region 208 is set as the tracking position. In the next tracking process, this tracking position is preferably set as the start position of the tracking process. As a result, the time required for the tracking process can be reduced.

  A tracking processing circuit 1306 as a second tracking position detection unit performs tracking processing using color information of image data as second feature information. The tracking process using this color information will be briefly described. In the tracking process using color information, a tracking color area that is an area that can be determined to be the same color as the color set in the evaluation image data is searched. As shown in FIG. 3A, when the position 302 of the subject is designated in the (N-1) frame, the color information of the position 302 in the evaluation image data is acquired. Then, an area having the same color information as the position 302 is searched for using the position 302 as the start position of the tracking process. Specifically, the color information is sequentially acquired from the start position toward the periphery from the start position, and if it can be determined that the acquired color information is the same as the color information at the position 302, the acquired color information is included in the area. When it cannot be determined that the information is the same as the color information at the position 302, it is not included in the area. When the tracking color area is searched in this way, for example, in the case of a monochromatic subject as shown in FIG. 3A, a rectangular area 304 inscribed in the subject becomes the tracking color area. Further, the tracking position recorded in the tracking position log area is, for example, the barycentric position of the tracking color area 304 (same as the position 302 in the example of FIG. 3A). In the next tracking process, this tracking position is set as the starting position of the tracking process.

  As an example of the N frame tracking process, the tracking position 302 of the (N-1) frame is tracked within a predetermined search range of the N frame image data stored as the reference image data shown in FIG. An area that can be determined to be the same color as the color of the tracking color area 304 is sequentially searched from the periphery of the tracking position 302 as the processing start position. In the example of FIG. 3B, the area 306 is a tracking color area. In the example of FIG. 3B, the center of gravity position 308 is set as the tracking position, and the tracking position 308 is recorded in the tracking position log area. Also, information indicating the range of the tracking color area 306 (for example, the positions of the four corners) is recorded in the tracking color area log area.

  A wrinkle detection circuit 1307 detects a wrinkle of a subject (person) in image data. Here, the wrinkle detection process will be briefly described. In the wrinkle detection process, the correlation amount between the image data obtained in each frame and the face parts 402, 404, and 406 as shown in FIG. The face part 402 is image data corresponding to a shadow pattern around a person's nose, the face part 404 is image data corresponding to a shadow pattern around a person's eye, and the face part 406 is a person This is image data corresponding to the shadow pattern around the mouth. The amount of correlation between the image processing data and the eyelid parts 402, 404, and 406 is maximized when a predetermined arrangement indicating a person's face is obtained as shown in FIG. At this time, it is assumed that a wrinkle exists in a region 408 including the wrinkle parts 402, 404, and 406. Note that the size of the face parts 402, 404, and 406 may be changed according to a preset search face size. Here, in FIG. 4B, the heel region is a rectangular region, but it may be a circular region.

A color information acquisition circuit 1308 as an example of a feature information acquisition unit acquires the color information of the tracking position obtained by the tracking processing circuit 1305.
The memory control circuit 1309 is an interface that performs control for the CPU 1301 and the like to access the RAM 118, the recording medium 128, and the ROM 134.

The operation unit 132 is various operation members operated by the user. The operation unit 132 includes, for example, a release button, a moving image button, a mode dial, a selection key, a power button, and the like.
The release button button has a 1st release switch and a 2nd release switch. The 1st release switch is a switch that is turned on when the user half-presses the release button. When the 1st release switch is turned on, a shooting preparation operation such as AF processing is performed. The 2nd release switch is a switch that is turned on when the user fully presses the release button. When the 2nd release switch is turned on, an exposure operation for still image shooting is performed.
The moving image button is an operation member for instructing the start or end of moving image shooting. When the moving image button is pressed by the user, the moving image shooting process is started. Further, when the moving image button is pressed during the moving image shooting process, the moving image shooting process is ended.
The mode dial is an operation member for selecting shooting settings of the imaging apparatus. In the present embodiment, for example, a still image shooting mode and a moving image shooting mode can be selected as shooting settings of the imaging apparatus. The still image shooting mode is a shooting setting for shooting a still image. The moving image shooting mode is a shooting setting for shooting a moving image.
The selection key is an operation member for selecting or determining an item on the menu screen, for example. When the selection key is operated by the user, an item is selected or determined on the menu screen.
The power button is an operation member for turning on or off the power of the imaging apparatus. When the power button is operated by the user, the imaging apparatus 100 is activated and becomes operable. If the power button is operated while the imaging apparatus is activated, the imaging apparatus 100 enters a power saving standby state.

  The ROM 134 stores program codes for the CPU 1301 to execute various processes. In addition, the ROM 134 stores various control parameters such as control parameters necessary for the operation of the photographing optical system 102, the aperture 106, the image sensor 114, and the like, and control parameters necessary for image processing in the image processing circuit 1304. ing. Furthermore, the ROM 134 also stores heel part data used for heel detection in the heel detection circuit 1307, data for displaying a tracking frame, and the like.

Next, the operation of the imaging apparatus according to the present embodiment will be described. FIG. 5 is a flowchart showing the shooting operation of the imaging apparatus 100. The CPU 1301 reads the necessary program code from the ROM 134 and controls the operation of FIG.
In step S <b> 100, the CPU 1301 determines whether the current shooting setting of the imaging apparatus 100 is the still image shooting mode. As described above, the shooting setting is set by the mode dial.

  When it is determined in S100 that the shooting setting is the still image shooting mode, in S102, the CPU 1301 starts a live view operation. As a live view operation, the CPU 1301 controls the shutter drive mechanism 112 to release the shutter 110, and then the CPU 1301 controls the image sensor IF circuit 116 to start imaging by the image sensor 114. Thereafter, the CPU 1301 inputs the image data stored in the work area of the RAM 118 as a result of imaging by the imaging device 114 to the image processing circuit 1304 and performs image processing for live view display. Subsequently, the CPU 1301 inputs image data that has been subjected to image processing for live view display to the display element driving circuit 122 and causes the display element 120 to display an image. By repeatedly executing such a display operation, the subject image is displayed as a moving image. By this moving image display, the user can observe the subject.

In step S104, the CPU 1301 determines whether the first release switch is turned on. The CPU 1301 continues the live view operation until it is determined in S104 that the 1st release switch is turned on.
If it is determined in S104 that the first release switch is turned on, in S106, the CPU 1301 performs a release AF process. In the release AF, the focus lens is driven to the in-focus position by scan driving. In the scan drive, the CPU 1301 controls the focus adjustment mechanism 104 to evaluate the focus evaluation values sequentially calculated by the AF control circuit 1302 while driving the focus lens in one direction within a predetermined scan range. Then, the CPU 1301 stops the driving of the focus lens at the lens position where the contrast is maximized as a result of the evaluation of the focus evaluation value. Such scan driving is performed when the difference between the position of the focus lens before AF and the in-focus position is large.

  In step S108, the CPU 1301 controls the display element driving circuit 122 to display a tracking frame on the display element 120. Here, the tracking frame is displayed at the position of the tracking target on the screen of the display element 120. For example, the subject focused by the release AF may be set as a tracking target, and a tracking frame may be displayed on the subject. When a wrinkle is detected by the wrinkle detection circuit 1307, a tracking frame is displayed on the face. You may make it let it. Furthermore, when a subject displayed on the screen of the display element 120 is designated by the touch panel 124, a tracking frame may be displayed on the subject. Thus, in the present embodiment, the CPU 1301, the AF control circuit 1302, the eyelid detection circuit 1307, the touch panel 124, and the like function as an example of the tracking target setting unit.

In S110, the CPU 1301 performs a tracking process. Details of this tracking process will be described later.
In step S112, the CPU 1301 performs AF processing so that the subject at the tracking position is focused, and performs AE processing so that exposure of the subject at the tracking position is appropriate.
In the AF processing after the tracking processing, the focus lens is driven to the in-focus position by scan driving or wobbling driving. In wobbling driving, the CPU 1301 determines whether or not the focus evaluation value calculated by the AF control circuit 1302 when the focus lens is driven has increased with respect to the focus evaluation value at the previous lens position. The CPU 1301 finely drives the focus lens in the same direction as the previous time when the focus evaluation value increases, and finely drives the focus lens in the opposite direction to the previous time when the focus evaluation value decreases. Such an operation is repeated at high speed to gradually drive the focus lens to the in-focus position.

  In the AE process, the CPU 1301 sets the aperture of the aperture 106 (aperture) during main exposure to set the luminance of the subject at the tracking position calculated by the AE control circuit 1303 to a predetermined appropriate amount (appropriate exposure amount). Value), the shutter 110 opening time (shutter speed value) is calculated.

  In step S114, the CPU 1301 determines whether the 2nd release switch is turned on. If it is determined in S114 that the 2nd release switch is not turned on, the CPU 1301 executes processing after the tracking processing in S110. In this way, in the still image shooting mode, the tracking process is continued until the 2nd release switch is turned on.

In S114, when it is determined that the 2nd release switch is turned on, in S116, the CPU 1301 controls the display element driving circuit 122 to hide the tracking frame.
In step S <b> 118, the CPU 1301 performs processing for recording still image data on the recording medium 128. At this time, the CPU 1301 controls the shutter driving mechanism 112 to close the shutter 110. Thereafter, the CPU 1301 controls the aperture driving mechanism 108 to narrow the aperture 106 to the previously calculated aperture value. Subsequently, the CPU 1301 controls the shutter driving mechanism 112 to perform imaging (exposure) by the imaging element 114 while opening the shutter 110 for the previously calculated opening time. Thereafter, the CPU 1301 processes still image data obtained via the image sensor 114 in the image processing circuit 1304. The CPU 1301 adds a header to the still image data processed by the image processing circuit 1304 to generate a still image file, and records the generated still image file on the recording medium 128.
In S120, the CPU 1301 adds data indicating the tracking position obtained as a result of the tracking processing in S110 to the still image file previously recorded on the recording medium 128. Thereafter, the CPU 1301 ends the operation shown in FIG.

If it is determined in S100 that the shooting setting is the moving image shooting mode, in S122, the CPU 1301 starts a live view operation.
In step S124, the CPU 1301 determines whether the moving image button has been turned on. Until it is determined in S124 that the moving image button has been turned on, the CPU 1301 continues the live view operation.

In S124, when it is determined that the moving image button is turned on, in S126, the CPU 1301 controls the display element driving circuit 122 to display the tracking frame on the display element 120.
In S128, the CPU 1301 performs a tracking process. Details of the tracking process will be described later.

  In step S130, the CPU 1301 performs AF processing so as to focus on the subject at the tracking position, and performs AE processing so that the exposure of the subject at the tracking position is appropriate. In the AF process in S130, the focus lens is driven to the in-focus position by wobbling driving.

  In step S <b> 132, the CPU 1301 performs processing for recording moving image data on the recording medium 128. At this time, the CPU 1301 controls the aperture driving mechanism 108 to narrow the aperture 106 to the aperture value calculated in the AE process. Subsequently, the CPU 1301 causes the image sensor 114 to perform imaging (exposure) for a time corresponding to the shutter speed value calculated in the AE process. After the exposure is completed, the CPU 1301 generates a moving image file and records it on the recording medium 128. Further, the CPU 1301 processes the moving image data obtained via the image sensor 114 in the image processing circuit 1304, and records the moving image data processed in the image processing circuit 1304 in a moving image file.

In S134, the CPU 1301 simultaneously records data indicating the tracking position obtained as a result of the tracking process in S128 together with the moving image file previously recorded on the recording medium 128.
In step S136, the CPU 1301 determines whether the moving image button has been turned off. If it is determined in S136 that the moving image button is not turned off, the CPU 1301 executes processing after the tracking processing in S128. Thus, in the moving image shooting mode, the tracking process and the recording of moving image data are continued until the moving image button is turned off.

If it is determined in S136 that the moving image button has been turned off, in S138, the CPU 1301 controls the display element driving circuit 122 to hide the tracking frame. Thereafter, the CPU 1301 ends the operation shown in FIG.
Hereinafter, tracking processing as a tracking method according to the present embodiment will be described with reference to FIGS. In the present embodiment, tracking of a tracking target is performed using a plurality of tracking processes in combination. FIG. 6 is a flowchart showing a tracking process as a tracking method according to an embodiment of the present invention. 7 and 8 are timing charts showing a tracking process as a tracking method according to an embodiment of the present invention.

  “Image data” shown in FIGS. 7 and 8 indicates “RAW data”, and “Image processing data” is data generated by performing the above-mentioned “special processing” on RAW data (for example, YC Data and image data obtained by combining a plurality of frames into one frame). Each graph shown in FIGS. 7 and 8 includes, in order from the top, “ON / OFF period of vertical synchronizing signal VD”, “exposure period of image sensor 114”, “RAW data generation period”, “ “Image processing data generation period”, “Tracking process period using first feature information (luminance information)”, “Tracking process period using second feature information (color information)”, “Face part of subject” “Tracking process period to be used”, “Determination of which tracking process is used and determination period of reliability of tracking position”, and “Update period of display position of tracking frame” are shown.

  As shown in FIGS. 7 and 8, the image sensor 114 starts exposure in synchronization with the vertical synchronization signal VD from the CPU 1301, and performs exposure for a preset period (one frame). That is, the CPU 1301 controls the image sensor IF circuit 116 with the vertical synchronization signal VD and executes image capturing with the image sensor 114. Then, the CPU 1301 captures the image data obtained in the imaging element IF circuit 116 into the RAM 118 by imaging with the imaging element 114.

  Note that the image data obtained by the first tracking process is evaluated image data. For this reason, the CPU 1301 captures the image data obtained in the first tracking process into the evaluation image area of the RAM 118. Further, the image data obtained by the second and subsequent tracking processing is referred to as reference image data. For this reason, the CPU 1301 captures the image data obtained in the second and subsequent tracking processing into the reference image area of the RAM 118.

  In the initial tracking process, only the evaluation image data has been acquired, so the processes of S203 to S219 and S222 to S232 are omitted. In the following description, the description will be continued on the assumption that both the evaluation image data and the reference image data have been acquired.

  In step S201, the CPU 1301 determines whether the time required for image processing by the image processing circuit 1304 is equal to or greater than a predetermined threshold. That is, it is determined whether “image processing time ≧ threshold”. For example, when the imaging apparatus 100 is set to the “second image generation mode” and “special processing” is performed, the tracking performance may be deteriorated. Processing time ≧ threshold ”. When the imaging apparatus 100 is set to the “first image generation mode”, “special processing” by the image processing circuit 1304 is not performed, and “image processing time <threshold”.

  By switching the subsequent tracking processing steps based on the determination result in S201, the CPU 1301 causes deterioration in tracking performance, for example, when the imaging apparatus 100 is set to the “second image generation mode”. Even if there is a possibility of occurrence, an appropriate tracking process can be executed.

  In the case of “image processing time ≧ threshold”, as shown in FIG. 8, when the result of the tracking process by face detection is included, the result determined by “tracking result determination (tracking position setting step in the flowchart of FIG. 6)” The final tracking position cannot be used as the start position in the next tracking process using luminance information / color information. In view of this, when the face part of the subject is detected, the tracking process using only the face part is executed by the series of processes of S201 to S206, and when the face part of the subject is not detected, A tracking process using only luminance information / color information is executed by a series of processes from S211 to S219. Details will be described below.

  If it is determined in S201 that the image processing time by the image processing circuit 1304 is equal to or greater than the predetermined threshold, the CPU 1301 acquires “image processing data” in S202. This “image processing data” is converted into image data (RAW data) based on a signal exposed by the image sensor 114 in synchronism with the vertical synchronization signal VD during the generation of the frame immediately before that frame by the image processing circuit 1304. This is image data generated by applying.

In step S <b> 203, the CPU 1301 executes detection processing for the eyelid part of the subject by the eyelid detection circuit 1307, and determines in step S <b> 204 whether a face part is detected.
If it is determined in S204 that a face part has been detected, the CPU 1301 calculates a tracking position (for example, the gravity center position of the face part region) from the face part in S205, and performs tracking processing using the face part. Since the tracking process using the face part has been described above, the description thereof will be omitted.

  In S206, the CPU 1301 records the final tracking position of the reference image data calculated in S205 in the tracking log area of the RAM 118.

In S210, the CPU 1301 controls the display element driving circuit 122 to update the display position of the tracking frame to a position corresponding to the tracking position stored in S206. Thereafter, the CPU 1301 ends the tracking process of FIG.
If it is determined in S204 that no face part has been detected, the CPU 1301 acquires “image data (RAW data)” in S211. This “image data” is image data (RAW data) based on a signal exposed by the image sensor 114 in synchronization with the vertical synchronization signal VD during the generation of the frame immediately before the relevant frame.

  In S212 and S213, the CPU 1301 uses the tracking processing circuit 1305 to perform tracking processing using luminance information as the first tracking processing (detection of luminance information and calculation of the tracking position). Since the tracking process using the luminance information has been described above, the description thereof will be omitted. Note that the CPU 1301 stores the tracking position obtained as a result of the processing in S212 and S213 (tracking processing using luminance information) in the tracking log area of the RAM 118 as the first candidate tracking position.

  In S214, the CPU 1301 determines the reliability of the tracking position obtained in S213. For example, the reliability is determined from the contrast of the reference image data. Specifically, when the sum of differences between adjacent pixels in the tracking target area in the reference image data is equal to or less than a predetermined value, it is determined that there is reliability. The threshold for determining reliability can be set as appropriate.

  In S215 and S216, the CPU 1301 uses the tracking processing circuit 1306 to perform tracking processing using color information as the second tracking processing (detection of color information and calculation of the tracking position). Since the tracking process using the color information has been described above, the description thereof will be omitted. Note that the CPU 1301 stores the tracking position obtained as a result of the processing of S215 and S216 (tracking processing using color information) in the tracking log area of the RAM 118 as the second candidate tracking position.

  In S217, the CPU 1301 determines the reliability of the tracking position obtained in S216. For example, the reliability is determined from the saturation of the reference image data. Specifically, if the saturation at the tracking position of the reference image data is greater than or equal to a predetermined value, it is determined that the reference image data is reliable. The threshold for determining reliability can be set as appropriate.

  In step S218, the CPU 1301 performs tracking position determination processing. The tracking position determination process is a process for determining which of a tracking process result using luminance information and a tracking process result using color information is to be adopted. The tracking position determination process may be, for example, a determination process based on the reliability of the tracking position calculated based on each feature information.

  In S219, the CPU 1301 records only the final tracking position adopted based on the determination result in S218 in the tracking log area of the RAM 118. In the next tracking process, the final tracking position is set as the starting position of the tracking process in both the tracking process using luminance information and the tracking process using color information.

In step S210, the CPU 1301 controls the display element driving circuit 122 to update the display position of the tracking frame to a position corresponding to the tracking position stored in step S219. Thereafter, the CPU 1301 ends the tracking process of FIG.
By the way, in the case of “image processing time <threshold”, as shown in FIG. 7, even if the result of tracking processing by face detection is included, “tracking result determination (tracking position setting step in the flowchart of FIG. 6)” is determined. The final tracking position thus made can be used as a start position in the next tracking process using luminance information / color information. Accordingly, both the tracking process using the face part of the subject and the tracking process using the luminance information / color information are executed. Details will be described below.

  If it is determined in S201 that the image processing time by the image processing circuit 1304 is not equal to or greater than the predetermined threshold, the CPU 1301 acquires “image data (RAW data)” in S221. This “image data” is image data (RAW data) based on a signal exposed by the image sensor 114 in synchronization with the vertical synchronization signal VD during the generation of the frame immediately before the relevant frame.

  In S222 and S223, the CPU 1301 causes the tracking processing circuit 1305 to perform tracking processing using luminance information as the first tracking processing (detection of luminance information and calculation of the tracking position). Since the tracking process using the luminance information has been described above, the description thereof will be omitted. Note that the CPU 1301 stores the tracking position obtained as a result of the processing of S222 and S223 (tracking processing using luminance information) in the tracking log area of the RAM 118 as the first candidate tracking position.

In S224, the CPU 1301 determines the reliability of the tracking position obtained in S213. For example, the reliability is determined from the contrast of the reference image data. Since the specific determination method has been described above, the description thereof will be omitted.
In S225 and S226, the CPU 1301 uses the tracking processing circuit 1306 to perform tracking processing using color information as the second tracking processing (detection of color information and calculation of the tracking position). Since the tracking process using the color information has been described above, the description thereof will be omitted. Note that the CPU 1301 stores the tracking position obtained as a result of the processing of S225 and S226 (tracking processing using color information) in the tracking log area of the RAM 118 as the second candidate tracking position.

In S227, the CPU 1301 determines the reliability of the tracking position obtained in S216. For example, the reliability is determined from the saturation of the reference image data. Since the specific determination method has been described above, the description thereof will be omitted.
In step S228, the CPU 1301 acquires “image processing data”. This “image processing data” is converted into image data (RAW data) based on a signal exposed by the image sensor 114 in synchronism with the vertical synchronization signal VD during the generation of the frame immediately before that frame by the image processing circuit 1304. This is image data generated by applying.

In step S <b> 229, the CPU 1301 executes detection processing for the eyelid part of the subject by the eyelid detection circuit 1307.
In S230, the CPU 1301 calculates a tracking position (for example, the gravity center position of the face part region) from the face part, and performs a tracking process using the face part. Since the tracking process using the face part has been described above, the description thereof will be omitted.

  In step S231, the CPU 1301 performs tracking position determination processing. This tracking position determination process determines which one of the result of the tracking process using the luminance information, the result of the tracking process using the color information, and the result of the tracking process using the face part is adopted. It is processing. The tracking position determination process may be, for example, a determination process based on the reliability of the tracking position calculated based on each feature information, or priority may be given to the tracking process using a face part depending on conditions.

  In S232, the CPU 1301 records only the final tracking position adopted based on the determination result of S231 in the tracking log area of the RAM 118. In the next tracking process, the final tracking position is set as the starting position of the tracking process in both the tracking process using luminance information and the tracking process using color information.

As described above, according to the present embodiment, a tracking device capable of performing appropriate tracking processing even when degradation of tracking performance, such as when performing special image processing, can occur, and A tracking method can be provided.
In the above-described example, color information, luminance information, and a subject face part are cited as feature information used for the tracking process. However, for example, contour information is detected by so-called silhouette detection, and the tracking process is performed using this. Of course, you may do.

  By the way, since the display frame rate in the live view display changes according to the image processing time by the image processing circuit 1304, the frame rate related to the image display when the first image generation mode is set, and the second image It is preferable that the frame rate related to the image display when the generation mode is set is a different frame rate.

Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. Further, in the description of the operation described above, the operation is described using “first”, “next”, and the like for convenience. However, it does not mean that the operation is essential in this order.
In the embodiment described above, the process is switched according to the image processing time and the threshold, but the determination criterion for switching the tracking calculation process is not limited to this. In the following, variations of the determination criteria will be described as application examples.
As a first application example, the set image processing mode is used as a determination criterion for switching the tracking calculation, such as switching the tracking processing between an image processing mode that takes time to process and an image processing mode that does not take time. Good.
As a second application example, the tracking processing is simplified when the priority is given to image processing, and the image processing is simplified when priority is given to the tracking processing. It may be a standard.
As a third application example, for example, in the case of a subject suitable for tracking by luminance, priority is given to the luminance tracking processing, and other tracking processing is simplified, and in the case of a subject suitable for tracking by color, color is selected. The feature of the subject may be used as a criterion for switching the tracking calculation so that the tracking process is prioritized and other tracking processes are simplified.
As a fourth application example, for example, when tracking is performed during continuous shooting, accuracy is prioritized when low-speed continuous shooting is selected, and normal tracking processing is performed when high-speed continuous shooting is selected. For example, the camera setting may be used as a criterion for switching the tracking calculation.
As a fifth application example, when the focal length of the taking lens is short (wide angle of view), priority is given to accuracy and normal tracking processing is performed. When the focal length is long (angle of view is narrow), priority is given to speed. In order to simplify the tracking process, the focal length of the lens may be used as a criterion for switching the tracking calculation.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 102 ... Imaging optical system, 104 ... Focus adjustment mechanism, 106 ... Aperture, 108 ... Aperture drive mechanism, 110 ... Shutter, 112 ... Shutter drive mechanism, 114 ... Imaging element, 116 ... Imaging element interface (IF) Circuit: 118 ... RAM, 120 ... Display element, 122 ... Display element drive circuit, 124 ... Touch panel, 126 ... Touch panel drive circuit, 128 ... Recording medium, 130 ... System controller, 132 ... Operation unit, 134 ... ROM, 1301 ... CPU 1302: AF control circuit, 1303 ... AE control circuit, 1304 ... Image processing circuit, 1305, 1306 ... Tracking processing circuit, 1307 ... Haze detection circuit, 1308 ... Color information acquisition circuit, 1309 ... Memory control circuit

Claims (9)

A first image generation mode for generating first image data, which is image data obtained by photographing, and a second for generating the first image data and performing predetermined processing on the first image data. A second image generation mode for generating the image data; a drawing mode setting unit for setting any one of the image generation modes;
An image processing unit that processes the image data input to the tracking device according to the setting of the drawing mode setting unit;
A tracking detection unit that detects feature information about the image data processed by the image processing unit and detects a face part of the subject;
A tracking calculation unit that calculates a movement position of the tracking target based on the feature information or the face part;
A calculation switching unit that switches the calculation by the tracking calculation unit to a calculation different from the calculation executed when the first image generation mode is set when the second image generation mode is set;
A tracking device comprising: When the second image generation mode is set, the calculation switching unit switches the calculation by the tracking calculation unit according to whether or not the face part of the subject is detected by the tracking detection unit. The tracking device according to claim 1. When the tracking detection unit detects the face part of the subject, the calculation switching unit switches the calculation by the tracking calculation part to a calculation based on the detection result of the face part of the subject. 2. The tracking device according to claim 1, wherein when a part is not detected, the calculation performed by the tracking calculation unit is switched to a calculation based on the feature information. The tracking device according to claim 1, wherein the feature information is at least one of color information, luminance information, and contour information. The predetermined process includes a process of combining a plurality of frames of image data to generate the second image data of one frame, a process similar to an optical process or development performed on one frame of image data, and 1 The tracking device according to claim 1, wherein the tracking device is at least one of a plurality of second image data generated by cutting out part of image data of a frame. The frame rate for image display when the first image generation mode is set is different from the frame rate for image display when the second image generation mode is set. 2. The tracking device according to 1. 2. The tracking device according to claim 1, wherein when the second image generation mode is set, the tracking calculation is performed with priority over the predetermined image processing. When the second image generation mode is set, when the predetermined image processing is performed with priority over the tracking calculation, some of the tracking calculation processes are not executed. The tracking device according to claim 1. A first image generation mode for generating first image data, which is image data obtained by photographing, and a second for generating the first image data and performing predetermined processing on the first image data. One of the second image generation modes for generating the image data and the image generation mode, and
The input image data is processed according to the setting of the drawing mode setting unit,
Detecting feature information about the processed image data, and detecting a face part of the subject,
Based on the feature information or the face part, the movement position of the tracking target is calculated,
When the second image generation mode is set, the calculation performed by the tracking calculation unit is switched to a calculation different from the calculation performed when the first image generation mode is set.
The tracking method characterized by this.

Images