JP5771461B2 - TRACKING DEVICE, TRACKING METHOD, AND TRACKING PROGRAM - Google Patents

Description

  The present invention relates to a tracking device, a tracking method, and a tracking program for tracking a subject such as a person.

  As a technique for tracking a subject such as a person, Patent Document 1 is known as a technique for tracking a subject using at least two areas for tracking the subject. This Patent Document 1 has a first area and a second area for tracking a subject on image data acquired by imaging the subject, and always tracks the subject using the first area as a reference area. The second area is used to track the subject independently of the first area. For example, the detection of the face area in the first area and the movement of the tracking area as the second area are performed independently of each other to detect the face area. And the position of the tracking area based on at least one of the positions in the image data, the position of the face area is estimated and set, and even when the face area cannot be directly detected, tracking is performed independently of the face area. Disclosed is to estimate and set the position of the face area based on the position of the area. In Patent Document 1, when it is determined that the first area and the second area are associated with each other in a plurality of previous image data, the positional relationship between the first area and the second area is specified. Is disclosed.

JP 2009-268086 A

  However, in Patent Document 1, when it is determined that the first area and the second area are associated with each other in a plurality of previous image data, the positional relationship between the first area and the second area is specified. Therefore, when the face area cannot be directly detected, the first area cannot be associated with the second area, and the position of the face area is determined based on the position of the tracking area moved independently of the face area. It becomes difficult to estimate. In addition, when there are a plurality of subjects, when the positional relationship between the first region and the second region is specified, the first region and the second region may be associated with each other in the same subject. If the first region and the second region are associated with each other on the subject and the subject different from each other, there is a risk of estimating the position of the face region of the subject different from the target subject. It may be difficult to track the target subject.

  An object of the present invention is to provide a tracking device, a tracking method, and a tracking program that can reliably track a subject when the subject is tracked using at least two regions.

Tracking apparatus according to the principal aspect of the present invention, a face part detecting section for detecting a face portion of the subject from the image data that the object is captured, the position corresponding to the face portion in the said image data Color information and a color information detection unit for detecting a region corresponding to the color information , position information of the face on the image detected by the face detection unit, and the color information detection unit A reliability determination unit that determines the reliability of the color information of the face , a result detected by the face detection unit , color information detected by the color information detection unit, and an area corresponding to the color information; the said is judged in reliability determination unit reliability and tracking the face portion based on the face part tracking unit, a peripheral part other than the face portion associated with said face portion of said object scene body during The peripheral part tracking part to be tracked and the tracking of the face part by the face part tracking part are If not come, anda tracking switching unit for switching the tracking of the portion other than the face portion by the peripheral part tracking unit, the tracking switching unit, as the face portion from the position of the peripheral portion other than the face portion When the face part is detected at the estimated position or the color information corresponding to the face part is detected, the face part tracking unit returns to the face part tracking.

Tracking method according to the main aspect of the present invention, in an image data that the object is imaged to detect a face portion of the subject by the face part detecting unit, the position corresponding to the face portion in the said image data Color information and a region corresponding to the color information are detected by a color information detection unit, and the position information of the face on the image detected by the face detection unit is detected by the reliability determination unit, and the color information detection The reliability of the color information of the face portion detected by the face portion, the result detected by the face portion detection portion , the color information detected by the color information detection portion, and the area corresponding to the color information When, wherein it is determined by the reliability determination unit based on the reliability tracking the face portion by a face part tracking section, the peripheral portion other than the face portion associated with said face portion of the photographic material during Is tracked by the peripheral part tracking unit, and by the face tracking unit If you can not tracking of Kikao part, you switch to the tracking of the peripheral portion other than the face portion due to the peripheral part tracking unit, switching of the tracking, the face part from the position of the peripheral portion other than the face part When the face part is detected at the position estimated as or the color information corresponding to the face part is detected, the face part tracking unit returns to the face part tracking.

Tracking program according to the principal aspect of the present invention causes a computer, a face part detecting function of detecting a face portion of the subject from the image data that the object is captured, corresponding to the face portion in the said image data Color information detection function for detecting color information of a position to be detected and a region corresponding to the color information , position information of the face on the image detected by the face detection function, and detection by the color information detection function a reliability determining function of determining the reliability of the color information of the face portion that is, a result detected by the face part detecting function, corresponding to the color information and the color information detected by the color information detecting function a region, said judged by the reliability determining function and reliability and the face portion tracking function for tracking the face portion based on said other than the face portion associated with said face portion of the photographic material during peripheral sites track the peripheral site And tail function, when the face portion tracking function due to not be tracking the face portion, to realize a tracking switching function for switching the tracking of the peripheral portion other than the face portion by the surrounding site tracking function, the tracking switching function Is the position estimated as the face from the position of the peripheral part other than the face, or the face is detected when the color information corresponding to the face is detected. Return to the tracking of the face by the tracking function.

  According to the present invention, it is possible to provide a tracking device, a tracking method, and a tracking program that can reliably track a subject when the subject is tracked using at least two regions.

The block block diagram which shows one Embodiment of the imaging device provided with the tracking apparatus which concerns on this invention. The figure for demonstrating the effect | action of the face part detection by the face part detection circuit in the apparatus. The figure for demonstrating the effect | action of the tracking of a buttocks using the brightness | luminance information by the buttocks tracking circuit in the apparatus. The figure for demonstrating the effect | action of the tracking of a buttocks using the color information by the buttocks tracking circuit in the apparatus. The figure which shows the face part detection position with respect to the subject in the same apparatus, a face part tracking position, and a peripheral part tracking position. The figure which shows the face part tracking frame set with respect to the face in front of the face part in the same apparatus. The figure which shows the face tracking frame set with respect to the profile of the face in the same apparatus. The figure which shows the chest tracking frame for tracking a chest by the peripheral region tracking circuit in the same apparatus. The imaging | photography operation | movement flowchart in the same apparatus. The tracking process flowchart in the same apparatus.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block configuration diagram as an example of an imaging apparatus including a tracking device. The imaging apparatus 100 includes an imaging optical system 102, a focus adjustment mechanism 104, a diaphragm 106, a diaphragm driving mechanism 108, a shutter 110, a shutter driving mechanism 112, an imaging element 114, and an imaging element interface (IF) circuit 116. A RAM 118, a display element 120, a display element drive circuit 122, a touch panel 124, a touch panel drive circuit 126, a recording medium 128, a system controller 130, an operation unit 132, and a ROM 134.

The photographing optical system 102 condenses the light flux F from the subject on the light receiving surface of the image sensor 114. The photographing optical system 102 has a plurality of lenses such as a focus lens. The focus adjustment mechanism 104 includes a motor, a drive circuit thereof, and the like, and drives the focus lens in the photographing optical system 102 in the optical axis direction (the one-dot chain line direction in the drawing) according to the control of the CPU 1301 as a computer in the system controller 130. .
The diaphragm 106 is configured to be openable and closable, and adjusts the amount of the light beam F incident on the image sensor 114 via the photographing optical system 102. The aperture drive mechanism 108 has a drive mechanism for driving the aperture 106 and drives the aperture 106 according to the control of the CPU 1301 in the system controller 130.

  The shutter 110 adjusts the exposure time of the image sensor 114 by setting the light receiving surface of the image sensor 114 to a light shielding state or an exposure state. The shutter drive mechanism 112 has a drive mechanism for driving the shutter 110 and drives the shutter 110 according to the control of the CPU 1301 in the system controller 130.

  The image sensor 114 has a light receiving surface on which a subject such as a person condensed through the photographing optical system 102 and a light flux F from the background thereof are imaged. The light receiving surface of the image sensor 114 is configured by two-dimensionally arranging a plurality of pixels, and a color filter is provided on the light incident side of the light receiving surface. Such an image sensor 114 converts an image (subject image) corresponding to the light beam F formed on the light receiving surface into an electrical signal (hereinafter referred to as an image signal) corresponding to the light amount. Here, as the imaging device 114, imaging devices having various configurations such as a CCD system and a CMOS system are known. Various color filters such as a Bayer array are also known. In the present embodiment, the configuration of the image sensor 114 is not limited to a specific configuration, and image sensors having various configurations can be used.

  The image sensor IF circuit 116 drives the image sensor 114 according to the control of the CPU 1301 in the system controller 130. The image sensor IF circuit 116 reads out an image signal obtained by the image sensor 114 under the control of the CPU 1301 in the system controller 130, and performs CDS (correlated double sampling) processing or AGC (automatic gain) on the read image signal. Control) Analog processing such as processing is performed. Further, the image sensor IF circuit 116 converts the analog processed image signal into a digital signal (hereinafter referred to as image data).

The RAM 118 is, for example, an SDRAM, and has a work area, an evaluation image area, a reference image area, a tracking position log area, and a tracking color area log area as storage areas.
The work area temporarily stores each data including image data obtained by the image sensor IF circuit 116, for example, each data generated in each part of the imaging device 100.
The evaluation image area temporarily stores evaluation image data. This evaluation image data is image data of a frame including a subject to be tracked in the tracking process. In the tracking process, processing is performed so as to track the tracking target.
The reference image area temporarily stores reference image data. This reference image data is image data of a frame to be searched for a tracking target in the tracking process. In the tracking process, a search is performed in the reference image data.
The tracking position log area temporarily stores a tracking position log. The tracking position log is a log in which the tracking position obtained as a result of the tracking process is recorded. In the present embodiment, a tracking target is tracked using a plurality of tracking processes. Therefore, in the tracking position log, the tracking positions obtained by the respective tracking processes are individually recorded, for example, for the past 10 frames. In addition, the final tracking position adopted by the priority tracking position determination process described later is also recorded in the tracking position log.
The tracking color area log area temporarily stores a tracking color area log. The tracking color area log is a log in which the tracking color area obtained by the tracking process is recorded. In the present embodiment, the tracking color area is recorded individually for the past 10 frames, for example.

The display element 120 is, for example, a liquid crystal display (LCD), and displays various images such as an image for live view and an image recorded on the recording medium 128. The display element driving circuit 122 drives the display element 120 based on the image data input from the CPU 1301 of the system controller 130 and displays an image on the display element 120.
The touch panel 124 is integrally formed on the display screen of the display element 120, and detects a contact position of a user's finger or the like on the display screen. The touch panel drive circuit 126 drives the touch panel 124 and outputs a contact detection signal from the touch panel 124 to the CPU 1301 of the system controller 130. The CPU 1301 detects a contact operation on the display screen of the user from the contact detection signal, and executes processing according to the contact operation.

  The recording medium 128 is a memory card, for example, and records an image file obtained by a shooting operation. This image file is configured by adding a predetermined header to image data. In the header, data indicating shooting conditions, data indicating a tracking position, and the like are recorded as tag data.

  The system controller 130 controls the operation of the imaging apparatus 100, and includes a CPU 1301, an AF control circuit 1302, an AE control circuit 1303, an image processing circuit 1304, a face detection circuit 1305, a color information detection circuit 1306, A heel tracking circuit 1307, a peripheral part tracking circuit 1308, a tracking switching circuit 1309, and a memory control circuit 1310 are provided.

  The CPU 1301 includes functional blocks outside the system controller 130 such as the focus adjustment mechanism 104, the aperture driving mechanism 108, the shutter driving mechanism 112, the display element driving circuit 122, the touch panel driving circuit 126, and the like. Controls the operation of each circuit.

The AF control circuit 1302 controls contrast AF processing. Specifically, the AF control circuit 1302 extracts high-frequency components of the image data obtained by the image sensor IF circuit 116, and obtains an AF focus evaluation value by integrating the extracted high-frequency components. The CPU 1301 controls the focus adjustment mechanism 104 to bring the focus lens into focus while evaluating the contrast of the image data according to the focus evaluation value.
The AE control circuit 1303 controls the AE operation. Specifically, the AE control circuit 1303 calculates the subject brightness using the image data obtained by the imaging element IF circuit 116. The CPU 1301 calculates the aperture amount (aperture value) of the aperture 106 during exposure, the opening time of the shutter 110 (shutter speed value), the image sensor sensitivity, the ISO sensitivity, and the like according to the subject brightness.

  The image processing circuit 1304 performs various types of image processing on the image data obtained by the imaging element IF circuit 116. This image processing includes, for example, color correction processing, gamma (γ) correction processing, compression processing, and the like. The image processing circuit 1304 also performs decompression processing on the compressed image data.

  The face part detection circuit 1305 detects the face part of the subject from image data in which a subject such as a person is captured. This face detection circuit 1305 obtains the correlation amount between the image data obtained in each frame and each face part 402, 404, 406 as shown in FIG. The face part 402 is image data corresponding to a shadow pattern around a person's nose, the face part 404 is image data corresponding to a shadow pattern around a person's eye, and the face part 406 is a person This is image data corresponding to the shadow pattern around the mouth. The correlation amount between the image data and each of the heel parts 402, 404, and 406 is maximized when a predetermined arrangement indicating the face of a person is obtained as shown in FIG. At this time, it is assumed that a human heel exists in an area 408 including the heel parts 402, 404, and 406. Note that the size of each of the heel parts 402, 404, and 406 may be changed according to a preset size of the search face. Here, in FIG. 2B, the heel region is a rectangular region, but it may be a circular region.

  The color information detection circuit 1306 detects an area of color information such as skin color corresponding to the face portion detected by the face portion detection circuit 1305 in the image data. Further, the color information detection circuit 1306 detects a region of color information such as skin color corresponding to a peripheral part other than the face part in the subject detected by the peripheral part tracking circuit 1308 in the image data.

  The buttock tracking circuit 1307 detects the face based on the luminance information of the face detected by the face detection circuit 1305 or the color information area of the skin color corresponding to the face detected by the color information detection circuit 1306, for example. To track. The hip tracking circuit 1307 tracks the face by prioritizing the face detection by the face detection circuit 1305 over the detection of the skin color information area by the color information detection circuit 1306.

Here, tracking of the buttocks will be described. The heel tracking circuit 1307 performs a tracking process using the luminance information of the image data. For example, when the subject to be tracked is set in the (N-1) frame shown in FIG. 3A, the tracking process using the luminance information is performed by using the image data of the (N-1) frame as the evaluation image data. Is stored in the evaluation image area of the RAM 118. Image data within a predetermined range 202 including the tracking target of the evaluation image data is set as the reference image data. Thereafter, in the tracking process, a portion corresponding to the standard image data 202 is searched for in the reference image data of N frames shown in FIG.
Taking the N frame tracking process as an example, the buttock tracking circuit 1307 stores N frame image data as reference image data in the reference image area of the RAM 118. The heel tracking circuit 1307 searches for a portion of the reference image data corresponding to the reference image data 202 by obtaining a correlation amount between the image data in the predetermined search range 204 and the reference image data 202. To do. The correlation amount is determined from, for example, the sum of absolute differences between the standard image data and the reference image data. The sum of absolute differences is obtained by calculating and integrating the absolute value of the luminance difference for each pixel.

  For example, if the sum of absolute differences between the reference image data in the reference image data area 206 and the standard image data 202 shown in FIG. 3B is obtained, the reference image data area 206 and the standard image data 202 are clearly shown. It is different image data, and the sum of absolute differences is increased. On the other hand, if the sum of absolute differences between the reference image data area 208 and the standard image data 202 is obtained, the sum of absolute differences becomes small. Thus, the difference absolute value sum decreases as the correlation amount with the reference image data 202 increases. In the tracking process using luminance information, an area having the maximum correlation amount, that is, the minimum difference absolute value sum is searched from the reference image data. In the example of FIG. 2B, the area 208 is searched as a portion corresponding to the reference image data 202. In the tracking position log area, the position with the highest degree of matching in the area 208 is recorded as the tracking position. When there are a plurality of such positions with a high degree of coincidence, for example, a position close to the center of the region 208 is set as the tracking position. In the next tracking process, this tracking position is preferably set as the start position of the tracking process. As a result, the time required for the tracking process can be reduced.

  Also, the buttock tracking circuit 1307 performs tracking processing using the skin color information of the image data. In the tracking process using the skin color information, an area that can be determined to be the same color as the color set in the evaluation image data is searched for as a tracking color area. As shown in FIG. 4A, the buttock tracking circuit 1307 acquires the color information of the position 302 in the evaluation image data when the position 302 where the subject is located is designated in the (N-1) frame. Then, the buttock tracking circuit 1307 searches for an area having the same color information as the position 302, with the position 302 as the start position of the tracking process. Specifically, the buttock tracking circuit 1307 sequentially acquires the skin color information so that the position 302 moves from the start position to the periphery, and when it can be determined that the acquired color information is the same as the color information of the position 302 If the acquired color information cannot be determined to be the same as the color information at the position 302, it is not included in the region. When the tracking color area is searched in this way, for example, in the case of a monochromatic subject as shown in FIG. 4A, a rectangular area 304 inscribed in the subject becomes the tracking color area. The tracking position recorded in the tracking position log area is, for example, the center of gravity position of the tracking color area 304 (same as the position 302 in the example of FIG. 4A). In the next tracking process, this tracking position is set as the starting position of the tracking process.

  Taking the N frame tracking process as an example, the buttock tracking circuit 1307 (N−1) within a predetermined search range of the N frame image data stored as the reference image data shown in FIG. The tracking position 302 of the frame is set as the tracking processing start position, and an area that can be determined to be the same color as the tracking color area 304 is sequentially searched from the periphery of the tracking position 302 as the tracking color area. In the example of FIG. 4B, the area 306 is a tracking color area. In the example of FIG. 4B, the center of gravity position 308 is set as the tracking position, and the tracking position 308 is recorded in the tracking position log area. Also, information indicating the range of the tracking color area 306 (for example, the positions of the four corners) is recorded in the tracking color area log area.

  The peripheral part tracking circuit 1308 tracks peripheral parts other than the face part in the subject. Similar to the buttock tracking circuit 1307, the peripheral part tracking circuit 1308 performs a peripheral part tracking process using luminance information of image data and skin color information. The peripheral part tracking circuit 1308 tracks, for example, a torso part of a subject having a preset positional relationship from the face part, specifically, a chest part existing in the torso part as a part other than the face part. The positional relationship between the face part and the chest part is as follows: the lower part than the face part of the subject on the image data, that is, the lower part than the face part of the person is the neck part, the chest part, the abdomen part, and the leg part. The position is set such that the position below the chest by a preset distance is the chest. In the peripheral part tracking circuit 1308, the peripheral part is not limited to the face part in the subject but may be, for example, the neck part, the abdomen part, the leg part, or the arm part.

FIG. 5 shows a face detection position A, a face tracking position B, and a peripheral tracking position C for the subject. The face detection position A indicates the face detected by the face detection circuit 1305 and corresponds to, for example, the rectangular area 408 shown in FIG.
The face tracking position B indicates a position for tracking the face based on the skin color information area corresponding to the face detected by the color information detection circuit 1306. For example, FIG. 4 (a) (b) Corresponds to the barycentric position 302 or 308 of the tracking color area 304 shown in FIG. The face tracking position B is recorded in the tracking position log area as described above.
The peripheral part tracking position C indicates a peripheral part other than the face part in the subject to be tracked by the peripheral part tracking circuit 1308, for example, the chest. The peripheral portion tracking position C indicates the chest in the subject by a rectangular frame, for example. The peripheral portion tracking position C is not limited to a rectangular frame, but may be a circular frame.

  Note that the buttock tracking circuit 1307 tracks the face based on the face detected by the face detection circuit 1305 or the skin color information area corresponding to the face detected by the color information detection circuit 1306. Sometimes, for example, as shown in FIG. 6 or FIG. 7, a rectangular face tracking frame D1 is displayed on the display element 120, which is an LCD, for example. FIG. 6 shows the face tracking frame D1 set for the face in front of the face, and FIG. 7 shows the face tracking frame D1 set for the side face of the face. The face tracking frame D1 is not limited to a rectangular frame, but may be a circular frame. The face portion is tracked as a face portion even if it is a front face or a side face.

  The peripheral part tracking circuit 1308 displays, for example, a rectangular chest tracking frame D2 on the display element 120 such as an LCD as shown in FIG. 8, for example, when tracking the chest as a peripheral part other than the face in the subject. . The chest tracking frame D2 is not limited to a rectangular frame, and may be a circular frame.

When the face tracking circuit 1307 cannot track the face, the tracking switching circuit 1309 uses, for example, the tracking of the face by displaying the face tracking frame D1 as shown in FIG. 6 or FIG. Switching to a part other than the face, for example, chest tracking as shown in FIG. 8, a rectangular chest tracking frame D2 is displayed on the chest on the display element 120, for example, LCD as shown in FIG.
The tracking switching circuit 1309 detects a face part or a skin color color information corresponding to the face part at a position estimated as a face part from a part other than the face part, for example, a chest part position. From the tracking of the chest by the peripheral part tracking circuit 1308 as shown in FIG. 8 (display of the chest tracking frame D2) to the tracking of the face by the face tracking circuit 1307 as shown in FIG. 6 or 7 (of the face tracking frame D1). Return to display.
Note that the positional relationship between the face and the chest is a predetermined distance from the subject's face on the image data, i.e., below the face of the person is the neck, chest, abdomen, and legs. Since the position below the face by a preset distance is set to be the chest, it can be estimated that the face exists above the person's chest by a preset distance.

The tracking switching circuit 1309 stores in advance the skin color information corresponding to the face, and when the color information at the position estimated as the face matches the color information stored in advance, the skin color corresponding to the face Is detected, the process returns to face tracking by the face tracking circuit 1307.
The tracking switching circuit 1309 can track the face by the face tracking circuit 1307 when at least the face detection circuit 1305 does not detect the face and the color information detection circuit 1306 does not detect the skin color information area. Judge that there is no.
If the face switching circuit 1309 does not detect the face in the image data and cannot track the face based on the skin color information area, and the performance of the tracking deteriorates, the face switching circuit 1307 causes the face It is determined that the part cannot be tracked.

The memory control circuit 1310 is an interface that controls the CPU 1301 and the like to access the RAM 118, the recording medium 128, and the ROM 134.
The operation unit 132 is various operation members operated by the user. The operation unit 132 includes, for example, a release button, a moving image button, a mode dial, a selection key, a power button, and the like.
The release button has a 1st release switch and a 2nd release switch. The 1st release switch is a switch that is turned on when the user half-presses the release button. When the 1st release switch is turned on, a shooting preparation operation such as an AF process is performed. The 2nd release switch is a switch that is turned on when the user fully presses the release button. When the 2nd release switch is turned on, an exposure operation for still image shooting is performed.
The moving image button is an operation member for instructing the start or end of moving image shooting. When the moving image button is pressed by the user, the moving image shooting process is started. Further, when the moving image button is pressed during the moving image shooting process, the moving image shooting process is ended.
The mode dial is an operation member for selecting shooting settings of the imaging apparatus. In the present embodiment, for example, a still image shooting mode and a moving image shooting mode can be selected as shooting settings of the imaging apparatus. The still image shooting mode is a shooting setting for shooting a still image. The moving image shooting mode is a shooting setting for shooting a moving image.
The selection key is an operation member for selecting or determining an item on the menu screen, for example. When the selection key is operated by the user, an item is selected or determined on the menu screen.
The power button is an operation member for turning on or off the power of the imaging apparatus. When the power button is operated by the user, the apparatus 100 is activated and becomes operable. If the power button is operated while the imaging apparatus is activated, the apparatus 100 enters a power saving standby state.

The ROM 134 stores program codes for the CPU 1301 to execute various processes. The program code includes a tracking program for tracking the subject. This tracking program causes a CPU 1301 as a computer to detect a face part of a subject from image data obtained by imaging the subject, and a face correspondence to track a position corresponding to the face part in image data. A part detection function, a face tracking function for tracking the face based on a result detected by the face detection function and a result detected by the face corresponding part detection function, and other than the face part in the subject The peripheral part tracking function for tracking the peripheral part of the face and the tracking switching function for switching to the tracking of the peripheral part other than the face part by the peripheral part tracking function when the face part tracking function cannot be performed by the face part tracking function are realized.
The tracking switching function is used when a face is detected at a position estimated from the position of a peripheral part other than the face, or when color information corresponding to the face is detected. Return to part tracking.

  The ROM 134 stores various control parameters such as control parameters necessary for the operation of the photographing optical system 102, the aperture 106, the image sensor 114, and the like, and control parameters necessary for image processing in the image processing circuit 1304. . Further, the ROM 134 also stores heel part data used for heel detection in the heel detection circuit 1307, data for displaying a tracking frame, and the like.

Next, the operation of the present apparatus configured as described above will be described with reference to the photographing operation flowchart shown in FIG.
The CPU 1301 reads necessary program codes from the ROM 134 and controls the operation of the apparatus 100.
In step S100, the CPU 1301 determines whether the current shooting setting of the apparatus 100 is the still image shooting mode. This shooting setting is set by the mode dial.
If the result of determination is that the shooting setting is still image shooting mode, the CPU 1301 starts the live view operation in step S102. As a live view operation, the CPU 1301 controls the shutter drive mechanism 112 to open the shutter 110, and then controls the image sensor IF circuit 116 to start imaging by the image sensor 114. Thereafter, the CPU 1301 inputs the image data stored in the work area of the RAM 118 as a result of imaging by the imaging device 114 to the image processing circuit 1304 and performs image processing for live view display. Subsequently, the CPU 1301 inputs image data that has been subjected to image processing for live view display to the display element driving circuit 122 and causes the display element 120 to display an image. By repeatedly executing such a display operation, the subject image is displayed as a moving image. By this moving image display, the user can observe the subject.

In step S104, the CPU 1301 determines whether the first release switch is turned on. As a result of this determination, the CPU 1301 continues the live view operation until it is determined that the first release switch is turned on.
When the first release switch is turned on, the CPU 1301 performs a release AF process in step S106. In this release AF, the focus lens is driven to the in-focus position by scan driving. In scan drive, the CPU 1301 controls the focus adjustment mechanism 104 to drive the focus lens in one direction within a predetermined scan range, and evaluates the focus evaluation values sequentially calculated by the AF control circuit 1302. The CPU 1301 stops the driving of the focus lens at the lens position where the contrast is maximized as a result of the evaluation of the focus evaluation value. Such scan driving is performed when the difference between the position of the focus lens before AF and the in-focus position is large.

  In step S108, the CPU 1301 controls the display element driving circuit 122 to display a tracking frame on the display element 120. Here, the tracking frame is displayed at the position of the tracking target on the screen of the display element 120. For example, a subject focused by the release AF may be set as a tracking target, and a tracking frame may be displayed on the subject. When the human eyelid is detected by the eyelid detection circuit 1307, for example, a tracking frame D1 may be displayed on the face as shown in FIG. Furthermore, when a subject displayed on the screen of the display element 120 is designated by the touch panel 124, a tracking frame may be displayed on the subject. Thus, in the present embodiment, the CPU 1301, the AF control circuit 1302, the eyelid detection circuit 1307, the touch panel 124, and the like function as an example of the tracking target setting unit.

In step S110, the CPU 1301 performs a tracking process for tracking the subject. This tracking process will be described later.
In step S112, the CPU 1301 performs AF processing so as to focus on the subject at the tracking position, and performs AE processing so that exposure of the subject at the tracking position is appropriate.
In the AF process after the tracking process, the focus lens is driven to the in-focus position by scan driving or wobbling driving. In wobbling driving, the CPU 1301 determines whether or not the focus evaluation value calculated by the AF control circuit 1302 when the focus lens is driven has increased with respect to the focus evaluation value at the previous lens position. The CPU 1301 finely drives the focus lens in the same direction as the previous time when the focus evaluation value increases, and finely drives the focus lens in the opposite direction to the previous time when the focus evaluation value decreases. Such an operation is repeated at high speed to gradually drive the focus lens to the in-focus position.
In the AE process, the CPU 1301 sets the aperture of the aperture 106 (aperture) during main exposure to set the luminance of the subject at the tracking position calculated by the AE control circuit 1303 to a predetermined appropriate amount (appropriate exposure amount). Value), the shutter 110 opening time (shutter speed value) is calculated.

  In step S114, the CPU 1301 determines whether or not the 2nd release switch is turned on. If it is determined that the 2nd release switch is not turned on, the CPU 1301 executes processing after the tracking processing in step S110. In this way, in the still image shooting mode, the tracking process is continued until the 2nd release switch is turned on.

When the 2nd release switch is turned on, the CPU 1301 controls the display element driving circuit 122 to hide the tracking frame D1 in step S116.
In step S <b> 118, the CPU 1301 performs processing for recording still image data on the recording medium 128. At this time, the CPU 1301 controls the shutter driving mechanism 112 to close the shutter 110. Thereafter, the CPU 1301 controls the aperture driving mechanism 108 to narrow the aperture 106 to the previously calculated aperture value. Subsequently, the CPU 1301 controls the shutter driving mechanism 112 to perform imaging (exposure) by the imaging element 114 while opening the shutter 110 for the previously calculated opening time. Thereafter, the CPU 1301 processes still image data obtained via the image sensor 114 in the image processing circuit 1304. The CPU 1301 adds a header to the still image data processed by the image processing circuit 1304 to generate a still image file, and records the generated still image file on the recording medium 128.
In step S120, the CPU 1301 adds data indicating the tracking position obtained as a result of the tracking process in step S110 to the still image file previously recorded on the recording medium 128. Thereafter, the CPU 1301 ends the operation according to the flowchart showing the photographing operation shown in FIG.

  On the other hand, when it is determined in step S100 that the shooting setting is the moving image shooting mode, the CPU 1301 starts a live view operation in step S122.

If it is determined that the moving image button, which is an instruction to turn on moving image shooting, is turned on, the CPU 1301 controls the display element driving circuit 122 in step S126 to cause the display element 120 to display a face portion as shown in FIG. To display the tracking frame D1.
In step S128, the CPU 1301 instructs the face detection circuit 1305, the face tracking circuit 1307, and the peripheral part tracking circuit 1308 to start tracking processing. This tracking process will be described later.
In step S130, the CPU 1301 instructs the AF control circuit 1302 to perform AF processing to focus on the subject at the tracking position, and instructs the AE control circuit 1303 to perform AE processing so that the subject at the tracking position is properly exposed. To do. In the AF process at this time, the focus lens of the photographing optical system 102 is driven to the in-focus position by wobbling driving.

  In step S <b> 132, the CPU 1301 performs processing for recording moving image data on the recording medium 128. At this time, the CPU 1301 controls the aperture driving mechanism 108 to narrow the aperture 106 to the aperture value calculated in the AE process. Subsequently, the CPU 1301 causes the image sensor 114 to perform imaging (exposure) for a time corresponding to the shutter speed value calculated in the AE process. After the exposure is completed, the CPU 1301 generates a moving image file and records it on the recording medium 128. Further, the CPU 1301 processes the moving image data obtained via the image sensor 114 in the image processing circuit 1304, and records the moving image data processed in the image processing circuit 1304 in a moving image file.

In step S134, the CPU 1301 records data indicating the tracking position obtained as a result of the tracking process in step S128 in the moving image file previously recorded on the recording medium 128.
In step S136, the CPU 1301 determines whether or not the moving image button has been turned off. As a result of this determination, if it is determined that the moving image button is not turned off, the CPU 1301 executes processing after the tracking processing in step S128. Thus, in the moving image shooting mode, the tracking process and the recording of moving image data are continued until the moving image button is turned off.

If it is determined that the moving image button is turned off, the CPU 1301 controls the display element driving circuit 122 to hide the tracking frame D1 in step S138. Thereafter, the CPU 1301 ends the operation according to the flowchart showing the photographing operation shown in FIG.
Next, the tracking process according to the present embodiment will be described with reference to the tracking process flowchart shown in FIG.
In step S <b> 200, the CPU 1301 controls the image sensor IF circuit 116 to perform image capturing with the image sensor 114.
In step S <b> 201, the CPU 1301 captures the image data obtained in the imaging element IF circuit 116 into the RAM 118 by imaging with the imaging element 114. The CPU 1301 uses the image data obtained by the first tracking process as evaluation image data, and takes this image data into the evaluation image area of the RAM 118. Further, the CPU 1301 uses the image data obtained by the second and subsequent tracking processes as reference image data, and takes the image data obtained by the second and subsequent tracking processes into the reference image area of the RAM 118, respectively.

  In step S202, the CPU 1301 uses the face detection circuit 1305 to detect the face of the subject from image data obtained by capturing the subject such as a person. The face detection circuit 1305 obtains the correlation amount between the image data obtained in each frame and each face part 402, 404, 406 as described with reference to FIGS. A region where a person's buttocks exists is detected using the fact that the amount is maximized when a predetermined arrangement indicating the person's face is obtained.

In step S203, the CPU 1301 causes the peripheral part tracking circuit 1308 to display a peripheral part other than the face part in the subject, for example, a chest below a predetermined distance from the face part detected by the face part detection circuit 1305. Set as.
In step S204, the CPU 1301 performs a face tracking process using the luminance information of the image data by the face tracking circuit 1307 and performs a chest tracking process using the luminance information of the image data by the peripheral part tracking circuit 1308. . In the tracking process using luminance information, the subject to be tracked is set in the (N-1) frame for the face and chest as described above with reference to FIGS. In this case, the image data of the (N-1) frame is stored in the evaluation image area of the RAM 118 as evaluation image data. Image data within a predetermined range 202 including the tracking target of the evaluation image data is set as the reference image data. Thereafter, in the tracking process, a portion corresponding to the standard image data 202 is searched for in the reference image data of N frames shown in FIG.

Note that in the first tracking process, only the evaluation image data has been acquired, so the processes after step S204 are omitted. In the following description, the description will be continued on the assumption that both the evaluation image data and the reference image data have been acquired.
In step S <b> 205, the CPU 1301 stores each tracking position obtained as a result of each tracking process of the face and chest using luminance information in the tracking log area of the RAM 118.

  In step S206, the CPU 1301 performs a face tracking process using the color information of the image data by the face tracking circuit 1307 and performs a chest tracking process using the color information of the image data by the peripheral part tracking circuit 1308. . In the tracking process using color information, an area that can be determined to be the same color as the color set in the evaluation image data for the face and chest is searched for as a tracking color area. As shown in FIG. 4A, the buttock tracking circuit 1307 acquires color information of the position 302 in the evaluation image data when the position 302 where the subject is located is designated in the (N−1) frame. Then, an area having the same color information as that of the position 302 is searched for using this position 302 as the tracking process start position.

  In step S <b> 207, the CPU 1301 stores each tracking position of the face and chest obtained as a result of the tracking process using color information in the tracking log area of the RAM 118. The CPU 1301 stores the tracking color area obtained as a result of the tracking process using the color information in the tracking color area log area of the RAM 118.

In step S <b> 208, the CPU 1301 determines whether or not the face can be tracked by the tracking switching circuit 1309. Specifically, the tracking switching circuit 1309 determines whether or not the face is not detected in the image data and the face cannot be tracked based on the skin color information area, and the tracking performance is deteriorated. judge.
When the face is detected in the image data or the face can be tracked based on the skin color information area, it is determined that the face can be tracked. Whether tracking is possible based on the skin color information area is determined based on whether the skin color information stored as a tracking target exists in the image data.
When the face is not detected in the image data and the face cannot be tracked based on the skin color information area, it is determined that the face cannot be tracked.

  If the result of this determination is that there is face data in the image data and the face can be tracked based on the skin color information area, the CPU 1301 performs priority tracking position determination processing in step S209. This priority tracking position determination process is a process for determining which one of the result of the tracking process using luminance information and the result of the tracking process using color information is used for tracking a face part. Then, since the face portion can be tracked using the luminance information and the color information as described above, priority is given to the result of the tracking processing using the color information, for example.

  In step S210, the CPU 1301 determines the reliability of the tracking position employed as a result of the priority tracking position determination process. When the result of the tracking process using the luminance information is adopted, for example, the reliability is determined from the contrast of the reference image data. Specifically, when the sum of differences between adjacent pixels in the tracking target area in the reference image data is equal to or greater than a predetermined value, it is determined that there is reliability. Further, when the result of the tracking process using color information is adopted, the reliability is determined from the saturation of the reference image data, for example. Specifically, if the saturation at the tracking position of the reference image data is greater than or equal to a predetermined value, it is determined that the reference image data is reliable. These threshold values for determining reliability can be appropriately set based on simulation results or experimental data.

In step S <b> 211, the CPU 1301 records the final tracking position of the reference image data in the tracking log area of the RAM 118. In the next tracking process, the final tracking position is set as the starting position of the tracking process in both the tracking process using luminance information and the tracking process using color information. However, if it is determined in step S216 that the reliability is not reliable, the tracking position may not be recorded.
In step S212, the CPU 1301 controls the display element driving circuit 122 to update the display position of the face tracking frame D1 to a position corresponding to the tracking position stored in step S218, for example, as shown in FIG. .

On the other hand, as a result of the determination in step S208, if there is no face data in the image data, the face cannot be tracked based on the skin color information area, and the tracking performance deteriorates. The CPU 1301 determines that the face cannot be tracked.
The CPU 1301 moves to step S213, and the tracking switching circuit 1309 switches from tracking the face to tracking by the peripheral part tracking circuit 1308 in the peripheral part. The peripheral part is a position estimated by the CPU 1301 from the face detection result and statistical data acquired in advance. Specifically, the position of the face is used as a reference, and the position below the reference position by the distance obtained according to the statistical data is set as the position of the face periphery. Here, the statistical data stored in the ROM 134 is statistical data from the position of the face portion measured at a certain fixed distance to the position of the face peripheral portion. Therefore, when the distance from the imaging device to the tracking target is different from the distance when the statistical data is obtained, the position of the face peripheral portion is corrected according to the distance difference. The distance from the imaging device to the tracking target is calculated from, for example, the size of the face (ratio of the size of the face on the screen). Then, the value of the statistical data is corrected according to this distance, and the position of the peripheral portion of the face on the actual image data is calculated. The peripheral part tracking circuit 1308 performs a peripheral part tracking process using luminance information and color information of image data.

  In step S214, the CPU 1301 performs priority tracking position determination processing. This priority tracking position determination process is a process for determining which one of the result of the tracking process using luminance information and the result of the tracking process using color information is to be used for tracking the peripheral part. Specifically, the priority tracking position is determined from the positional relationship of the tracking position calculated from the luminance information and the color information and the color information at the tracking position calculated from the luminance information and the color information.

  In step S215, the CPU 1301 determines the reliability of the tracking position adopted as a result of the priority tracking position determination process. When the result of the tracking process using the luminance information is adopted, for example, the reliability is determined from the contrast of the reference image data. Specifically, when the sum of differences between adjacent pixels in the tracking target area in the reference image data is equal to or greater than a predetermined value, it is determined that there is reliability.

In step S <b> 211, the CPU 1301 records the final tracking position of the reference image data in the tracking log area of the RAM 118. In the next tracking process, this final tracking position is set as the start position of the tracking process in the tracking process using luminance information. However, if it is determined in step S216 that the reliability is not reliable, the tracking position may not be recorded.
In step S212, the CPU 1301 controls the display element driving circuit 122 to update the display position of the chest tracking frame D2 to a position corresponding to the tracking position stored in step S211 as shown in FIG. 6 or FIG.

The CPU 1301 executes steps S201 to S207. In step S208, the tracking switching circuit 1309 estimates whether or not the face can be tracked, that is, the face is estimated from a part other than the face, for example, the position of the chest. It is determined whether or not the face portion is detected or the skin color information corresponding to the face portion is detected.
As a result of the determination, for example, if the face is not detected at the position estimated as the face from the position of the chest, or if the skin color information corresponding to the face is not detected, the CPU 1301 proceeds to step S213. The tracking of the chest is continued by the tracking switching circuit 1309. As described above, the peripheral part tracking circuit 1308 performs the chest tracking process using the luminance information and the color information of the image data.

On the other hand, as a result of the above determination, when the face is detected at the position estimated as the face from the position of the chest or the skin color information corresponding to the face is detected, the CPU 1301 detects the peripheral part tracking circuit 1308. Return from tracking the chest to tracking the face by the face tracking circuit 1307. The tracking switching circuit 1309 stores the color information of the skin color corresponding to the face in advance, and corresponds to the face when the color information at the position estimated as the face matches the color information stored in advance. Suppose that color information of the skin color to be detected is detected.
The CPU 1301 executes steps S209 to S211 described above, and in step S212, the chest tracking (display of the chest tracking frame D2) as shown in FIG. 8 by the peripheral part tracking circuit 1308 as shown in FIG. Returning to the face tracking (display of the face tracking frame D1) by the face tracking circuit 1307 as shown in FIG.
Thereafter, the CPU 1301 ends the tracking process according to the tracking process flowchart shown in FIG.

Thus, according to the above-described embodiment, when tracking cannot be continued in the face portion of the subject, the subject is tracked by switching to, for example, chest tracking as a peripheral region other than the face portion. Thus, for example, when tracking a subject using at least two areas of the face and the chest, for example, even if the tracking result using the face as a reference area is not good, the area such as the chest is used as the reference area. And the subject can be reliably tracked using the region such as the chest.
Also, in the state where the subject is being tracked using the region such as the chest as a reference region, the face is detected at a position estimated as the face from the position of the chest or color information corresponding to the face is detected. Then, tracking using the face can be performed again.

Switching from face tracking to chest tracking, for example, indicates that there is no face data in the image data and the face cannot be tracked based on the skin color information area, and the tracking performance deteriorates. Since it is a condition, it is possible to switch to tracking the chest when the face tracking is completely impossible.
The return from the tracking of the chest to the tracking of the face is performed on the condition that the face is detected at the position estimated as the face from the position of the chest or that the skin color information corresponding to the face is detected. Therefore, it is possible to return to following the face when the face is completely tracked.

  The face portion and the chest portion are set to have a relationship of being a chest portion below the face portion of the subject on the image data by a preset distance, and the face portion is above the chest portion by a preset distance. Since it is set to exist, for example, even if face tracking cannot be performed, switching to the chest tracking of the same person can be surely performed, and switching to the tracking of the chest of another person is not performed. Similarly, when returning from the tracking of the chest to the tracking of the face, it is possible to surely switch to the tracking of the face of the same person, without switching to the tracking of the face of another person. Thereby, even if the tracking target part is switched, tracking of the same person can always be continued.

In the above-described embodiment, the tracking of the face and the tracking of the chest are switched. However, the tracking is not limited to this, and the tracking of the face is set as the main tracking, and the tracking of the peripheral part is performed, for example, the tracking of the neck and the abdomen. Any one of the following tracking, leg tracking, and arm tracking may be added.
Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 100: Imaging device, 102: Imaging optical system, 104: Focus adjustment mechanism, 106: Aperture, 108: Aperture drive mechanism, 110: Shutter, 112: Shutter drive mechanism, 114: Image sensor, 116: Image sensor interface (IF) Circuit: 118: RAM, 120: Display element, 122: Display element drive circuit, 124: Touch panel, 126: Touch panel drive circuit, 128: Recording medium, 130: System controller, 132: Operation unit, 134: ROM, 1301: CPU 1302: AF control circuit, 1303: AE control circuit, 1304: Image processing circuit, 1305: Face detection circuit, 1306: Color information detection circuit, 1307: Hip part tracking circuit, 1308: Peripheral part tracking circuit, 1309: Tracking Switching circuit, 1310: Memory control circuit, 402, 404, 406: Part, 308: center-of-gravity position, A: the face part detection position, B: the face part tracking position, C: periphery tracking position, D1: rectangular tracking frame.

Claims (9)

A face part detecting section for detecting a face portion of the subject from the image data that the object is captured,
Color information at a position corresponding to the face in the image data and a color information detection unit for detecting a region corresponding to the color information ;
A reliability determination unit that determines the reliability of the position information of the face on the image detected by the face detection unit and the color information of the face detected by the color information detection unit;
Based on the result detected by the face detection unit , the color information detected by the color information detection unit, the area corresponding to the color information, and the reliability determined by the reliability determination unit A face tracking unit for tracking the face,
A peripheral part tracking unit for tracking the peripheral portion other than said face section associated with the face portion of the photographic material in,
When the face tracking unit cannot track the face part, a tracking switching unit that switches to tracking of a part other than the face part by the peripheral part tracking part;
Comprising
The tracking switching unit detects the face or detects the color information corresponding to the face at a position estimated as the face from the position of the peripheral part other than the face. A tracking device that returns to tracking of the face by the face tracking unit. The tracking switching unit stores color information corresponding to the face portion in advance, and when the color information at a position estimated as the face portion matches the color information stored in advance, The tracking device according to claim 1, wherein the tracking is returned to the tracking of the face by the face tracking unit when the corresponding color information is detected. The face portion tracking unit, rather than the detection of the area corresponding to the color information by the color information detecting unit, according to claim 1, characterized in that priority is given to detection of the face portion by the face part detecting section Tracking device. The tracking switching unit is configured to detect the face by the face tracking unit when at least the face detection unit does not detect the face and the color information detection unit does not detect the area corresponding to the color information. The tracking device according to claim 1, wherein it is determined that the tracking of the first track cannot be performed. The tracking switching unit, said face portion without data is in the image data, and to be able to track the face portion based on the area corresponding to the color information, the face by the face part tracking unit The tracking device according to claim 4, wherein it is determined that tracking of a part cannot be performed. The color information detecting unit, a tracking device according to any one of claims 1 to 5, characterized in that detecting the skin color as the color information. The surrounding site tracking unit tracking apparatus according to claim 1, characterized in that tracking the body portion of the object scene having a predetermined positional relationship from the face portion as the site other than the face portion. Detecting a face portion of the subject by the face part detecting section from an image data that the object is captured,
The color information detection unit detects color information at a position corresponding to the face portion in the image data and a region corresponding to the color information ,
The reliability determination unit determines the reliability of the position information of the face part on the image detected by the face part detection unit and the color information of the face part detected by the color information detection unit,
Based on the result detected by the face detection unit , the color information detected by the color information detection unit, the area corresponding to the color information, and the reliability determined by the reliability determination unit The face tracking unit tracks the face part,
Wherein the peripheral portion other than the face portion associated with said face portion of the photographic material during tracked by the peripheral part tracking unit,
If you can not tracking of the face part by the face part tracking unit, it switches to the tracking of the peripheral portion other than the face portion due to the peripheral part tracking unit,
The tracking switching is performed when the face part is detected or the color information corresponding to the face part is detected at a position estimated as the face part from the position of the peripheral part other than the face part. The tracking method returns to the tracking of the face by the face tracking unit . On the computer,
A face part detecting function of detecting a face portion of the subject from the image data that the object is captured,
A color information detection function for detecting color information at a position corresponding to the face portion in the image data and a region corresponding to the color information ;
A reliability determination function for determining the reliability of the position information of the face on the image detected by the face detection function and the color information of the face detected by the color information detection function;
Based on the result detected by the face detection function , the color information detected by the color information detection function, the area corresponding to the color information, and the reliability determined by the reliability determination function A face tracking function for tracking the face,
A surrounding site tracking function to track the peripheral portion other than the face portion associated with said face portion of said object scene body in,
If you can not track the face portion by the face portion tracking function, and tracking the switching function for switching the tracking of the peripheral portion other than the face portion by the surrounding site tracking function,
Realized ,
When the tracking switching function detects the face or the color information corresponding to the face at a position estimated as the face from the position of the peripheral part other than the face , Returning to the face tracking by the face tracking function,
Tracking program.

Images