JP2014063280A - Object tracking method and device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object tracking method and device and a program which are capable of improving tracking accuracy.SOLUTION: In the object tracking method detecting a specific object from a captured image acquired in time series and tracking the position of the specific object, a first template matching processing using an initial model is performed to the captured image inputted after the initial model is generated and when a first score indicating similarity indicates high similarity equal to or higher than a predetermined threshold value, a position searched by the first matching processing is adopted as a tracking position. On the other hand, when the first score indicates the similarity lower than the predetermined threshold value, a second template matching processing is performed by using the latest update model generated from the tracking position of the captured image of a preceding frame and the position searched by the second matching processing is adopted as the tracking position.

Description

  The present invention relates to an image processing technique for detecting a specific object (object) from a captured image and tracking (tracking) its position, and more particularly to an object tracking method, apparatus, and program using a template matching technique.

  An image processing technique for tracking an object using a template matching method is known (Patent Documents 1 to 3). The tracking method based on template matching uses a single reference template to match an unknown image, and the similarity to the reference template ("matching degree", "correlation degree", "matching score", etc.) Also search for the place with the highest. If the similarity exceeds a certain determination condition (predetermined threshold), the location is determined as the position of the object.

  Appearance information (initial model) at the start of tracking may be used as a reference template. In addition, the latest model (update model) based on the latest tracking result may be used as a reference template.

  A system that monitors a driver (driver) of a vehicle such as an automobile and detects the state of the driver such as falling asleep using a tracking technique based on template matching has been proposed (Patent Documents 4 to 6).

JP 2010-282537 A JP 2001-60265 A JP 2007-304852 A JP 2010-97379 A JP 2005-327072 A JP 2001-101386 A

  However, for example, when a person's eyes are set as a tracking target, the orientation of the face changes frequently. In addition, since the face approaches or moves away from the camera, the size (size) of the face component in the captured image changes. For this reason, the matching method using only the initial model (also referred to as “initial template”) generated from the appearance information at the start of tracking cannot sufficiently cope with changes in face orientation, perspective changes, and the like.

  On the other hand, when using an update model (also referred to as “update template”) generated from the latest tracking results as a reference template, it can cope with angular variations and perspective variations in the face direction, but misalignment (error) during tracking There is a theoretical drawback in that the position error gradually increases.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an object tracking method, apparatus, and program capable of solving the above-described problems and improving tracking accuracy.

  In order to achieve the above object, the following invention is provided.

  (First aspect): An object tracking method for detecting a specific object from a captured image acquired in time series and tracking its position, specifying an image portion of the object from an input captured image, An initial model generation step for generating an initial model indicating image features, and template matching processing using at least the initial model for the captured image input after the initial model is generated, and searching for the position of the object from the captured image An object search step, and an update model generation step of generating an update model from the image portion of the object at the tracking position specified by the object search step and storing the latest update model. The first template matching process using the initial model and A first matching processing step for calculating a first score indicating the similarity, a comparison step for comparing the first score obtained by the first matching processing step with a first threshold, and the latest stored As a result of the comparison by the second matching processing step of performing the second template matching process using the update model and calculating the second score indicating the degree of similarity with the update model and the comparison step, the first score is In the case of showing high similarity equal to or higher than the threshold value of 1, the matching position searched in the first matching processing step is adopted as the tracking position, while the first score shows similarity lower than the first threshold value. A tracking position determination step that employs the matching position searched in the second matching processing step as a tracking position.

  According to this aspect, the object search is performed by combining the first matching process using the initial model and the second matching process using the latest update model updated as needed. At that time, basically, the result of the first matching process by the initial model is prioritized, and the second matching process by the updated model is performed when a position equal to or greater than the first threshold value cannot be detected by the initial model. The result is adopted.

  With such a configuration, it is possible to deal with perspective variation and angle (rotation) variation while solving the problem of accumulation of tracking position errors.

  (Second Aspect): In the object tracking method according to the first aspect, after the initial model is generated, the first matching process step is preferentially performed on each captured image that is sequentially input, and the comparison by the comparison step is performed. As a result, when the first score shows high similarity equal to or higher than the first threshold, the second matching processing step can be omitted.

  That is, only when the first score shows a similarity lower than the first threshold, the second matching process can be performed.

  (Third Aspect): The object tracking method according to the first aspect or the second aspect includes a tracking stop determination step for determining whether to stop or continue tracking based on the tracking result of the object search step. be able to.

  (Fourth aspect): In the object tracking method according to the third aspect, the tracking stop determination step includes the tracking position searched by the object search step, the first score, the second score, and the size of the object. The tracking stop may be determined based on at least one information.

  (Fifth aspect): In the object tracking method according to the third aspect or the fourth aspect, when the tracking is stopped based on the determination in the tracking stop determination step, the initial model can be recreated.

  (Sixth aspect): In the object tracking method according to any one of the first aspect to the fifth aspect, a step of storing tracking result information about a plurality of frames of captured images continuous in time series as a history. Can be included.

  (Seventh aspect): In the object tracking method according to any one of the first to sixth aspects, a horizontal angle indicating a horizontal rotation angle of the object is calculated from the tracking position determined by the tracking position determining step. And an initial model update step of changing the initial model according to the angle calculated in the angle estimation step.

  (Eighth aspect): In the object tracking method according to the seventh aspect, a plurality of angle correspondence models corresponding to a plurality of rotation angles are created in advance based on the initial model generated in the initial model generation step. Including a step of selecting one angle correspondence model from a plurality of angle correspondence models based on the angle calculated in the estimation step, and storing the selected angle correspondence model as a new initial model instead of the initial model. Can do.

  (Ninth aspect): In the object tracking method according to any one of the first aspect to the eighth aspect, the initial model generation step uses a captured image using an object discriminator constructed by machine learning from a large number of images. An initial model is generated by detecting the image portion of the object from the image.

  (Tenth aspect): An object tracking device that detects a specific object from a captured image acquired in time series and tracks its position, specifies an image portion of the object from the input captured image, and An initial model generation unit that generates an initial model indicating image features, and a template matching process that uses at least the initial model for a captured image input after the initial model is generated, and searches for the position of the object from the captured image An object search unit; and an update model generation unit that generates an update model from the image portion of the object at the tracking position specified by the object search unit and stores the latest update model. Perform the first template matching process using, and the similarity with the initial model A first matching processing unit that calculates a first score to be shown, a comparison unit that compares a first score obtained by the first matching processing unit with a first threshold, and a stored latest update model A second matching processing unit that performs a second template matching process and calculates a second score indicating the degree of similarity with the update model, and as a result of comparison by the comparison unit, the first score is a first threshold value. The matching position searched by the first matching processing unit is employed as the tracking position when the similarity is equal to or higher than the second, and the second is when the first score is lower than the first threshold. A tracking position determination unit that employs a matching position searched by the matching processing unit as a tracking position.

  In the object tracking device of the tenth aspect, configurations similar to the matters shown in the second aspect to the ninth aspect can be appropriately combined. In this case, the element shown as "process" is specified as a means corresponding to this, or a function part performing the function.

  (Eleventh aspect): A program for causing a computer to execute each step of the object tracking method according to any one of the first to ninth aspects is provided.

  According to the present invention, it is possible to solve the problem of cumulative enlargement of tracking position error by the conventional tracking method, and achieve improvement in tracking accuracy. Further, even when a perspective variation or an angle (rotation) variation occurs, the position of the object can be specified and tracking can be performed.

The block diagram which shows the whole structure of the surveillance video system which concerns on embodiment of this invention Flow chart showing the flow of the whole dozing detection process The flowchart which shows the 1st example regarding the flow of the process of lock-on determination The flowchart which shows the 2nd example regarding the flow of a specific process of lock-on determination The flowchart which shows the flow of the eye tracking process by this embodiment. Flowchart illustrating an example of determination processing for performing tracking stop and return processing from stop State transition diagram showing transition relationships between three possible states during tracking Flow chart showing the flow of open / close determination processing Graph showing the relationship between face angle and detection score Flow chart showing the flow of face angle estimation processing and aside look determination processing Schematic diagram for explaining how to calculate the face angle (horizontal angle) Flow chart showing how to update the initial model Conceptual diagram of processing to generate an angle template corresponding to the face orientation angle Block diagram showing a configuration example of an image processing unit

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing the overall configuration of a surveillance video system according to an embodiment of the present invention. The monitoring video system 10 includes a camera 12 as an imaging device, an illumination device 14 that emits illumination light, and a control device 16. Although the detailed configuration of the camera 12 is not shown, a photoelectric conversion element (imaging device) such as a CCD (charge-coupled device) imaging element or a CMOS (Complementary Metal-Oxide Semiconductor) imaging element as in a general electronic imaging apparatus. And a required electronic circuit, which converts an optical image of a subject photographed through the lens unit 18 into an output electrical signal and outputs an imaging signal corresponding to the optical image.

  As the camera 12, an infrared camera having sensitivity in the infrared wavelength region can be used. Further, instead of the infrared camera, a color imaging camera equipped with a color imaging device including RGB color separation color filters may be used, or a monochrome camera that acquires a monochrome image may be used.

  The camera 12 is equipped with a signal processing circuit such as a gain control circuit, a sampling circuit, and an A / D converter in addition to a drive circuit for the image sensor. The analog image signal obtained from the image sensor is digital image data. And output from the camera 12. The imaging data output from the camera 12 is sent to the control device 16 by wired or wireless signal transmission means.

  The control device 16 includes an image processing unit 20 that processes captured image data obtained from the camera 12, a camera control unit 22 that controls the camera 12, and an illumination light control unit 24 that controls the illumination device 14.

  The image processing unit 20 functions as an object tracking device that detects a specific object from an input captured image and tracks its position. Details will be described later.

  The camera control unit 22 controls the start (ON) / stop (OFF) of the camera 12 and sets the imaging cycle (frame rate). The camera control unit 22 can also perform automatic exposure control (AE) and automatic focus adjustment control (AF) of the camera 12 in cooperation with the image processing unit 20.

  The illumination light control unit 24 controls the amount of light emission (illuminance adjustment, light emission time, light emission frequency, etc.) in addition to the lighting (ON) / light-off (OFF) control of the lighting device 14. The illuminating device 14 can be configured to include a plurality of light emitting units so that the driver who is the subject can irradiate illumination light from a plurality of directions. For example, a plurality of light emitting units can be arranged in a symmetrical relationship with respect to the camera 12. The illumination light control unit 24 may be configured to control a plurality of light emitting units simultaneously (for example, collectively with the same command signal), or may be configured to control each light emitting unit individually (independently).

  The illumination light control unit 24 controls the light amount of the illumination device 14 so as to obtain an image having a necessary brightness in cooperation with the camera control unit 22. It is also possible to configure as a system control unit in which the functions of the camera control unit 22 and the illumination light control unit 24 are integrated.

  The light emission source of the illumination device 14 is not particularly limited, and any appropriate means such as a lamp, a light-emitting diode (LED), a semiconductor laser, or the like can be employed. A configuration using LEDs is preferable from the viewpoints of light quantity control responsiveness, power saving, and the like, and the illumination device 14 of this example is a group of LEDs in which a plurality of LEDs are two-dimensionally arranged in order to realize a necessary illumination light irradiation range. Is used.

  When the camera 12 of the imaging unit is an infrared camera, an illumination device 14 that emits infrared rays is used. However, in the practice of the invention, the spectrum of the illumination light is not particularly limited. Instead of infrared light, white light may be used, a specific wavelength component distribution may be used, or an appropriate combination thereof may be used. An appropriate light source (spectrum) is selected in relation to the camera 12 and the image processing unit 20.

  The monitoring video system 10 of this example is mounted on a vehicle such as an automobile, for example, and is used as a driver monitoring system that detects a driver's state (sleeping or looking aside) by photographing the driver's face with the camera 12. it can.

  In the following description, a monitoring video system that tracks the position of the “eye” from the driver's face image captured by the camera 12 and senses a state such as falling asleep will be described as an example. In this case, the camera 12 captures the driver's face in the field of view through the lens unit 18, and the camera position and angle of view are adjusted so that the driver's face can be photographed from the front side in a normal driving state. Is done. The illumination device 14 is also installed at an appropriate location so that illumination light having a necessary illuminance can be applied to the driver's face (particularly, the region including both the left and right eyes).

  Imaging is continuously performed by the camera 12 at a constant time interval (for example, 60 frames / second), and captured image data is acquired from the camera 12 in time series. The image processing unit 20 sequentially processes each frame of the captured image acquired in time series from the camera 12. That is, the image processing unit 20 can process a moving image in real time.

<Overall processing flow>
FIG. 2 is a flowchart showing the flow of the whole dozing detection process in the image processing unit 20. FIG. 2 shows the flow of processing for image input of one frame. When an image of one frame is input to the image processing unit 20 (step S12), it is first determined whether or not the input image is an initial frame (step S14). If the input image is an initial frame (for example, the first frame from which captured image capture is started) (Yes in step S14), the process proceeds to step S16, and the lock on flag is turned off (OFF). ).

  The lock-on flag is a state identification code indicating whether or not both eyes of the driver who is the tracking target can be detected. When both eyes are detected, the lock-on flag is “ON”, and when both eyes are not detected, the lock-on flag is “OFF”. At the stage when the initial frame is input, since the driver's eyes have not been detected yet (the state is not being tracked), the lock-on flag is set to “OFF”.

  Next, lock-on determination processing is performed (step S18). The lock-on determination process is a process for detecting the positions of both eyes of the driver from the input image in order to monitor the eyes of the driver. Specifically, a discriminator for detecting both eyes opened by a person (referred to as a “binocular discriminator”) obtained by machine learning using the AdaBoost algorithm, and a discriminator for detecting an open monocular ( In combination with a “monocular discriminator”), the left and right eyes are detected. Details of the lock-on determination process will be described later with reference to FIGS. The binocular discriminator and the monocular discriminator each correspond to an “object discriminator”.

  When both eyes of the driver are detected by the lock-on determination process (step S18 in FIG. 2), the lock-on flag is set to “ON”.

  In step S20, it is determined whether or not the lock-on flag is ON. When it is confirmed that the lock-on flag is ON (Yes in step S20), processing for setting the tracking flag to ON (step S22) is performed. The tracking flag is a state identification code indicating whether or not the driver's eye as the tracking target is being tracked. If the tracking flag is set to ON in step S22, the processing for the one frame is terminated.

  On the other hand, if both eyes cannot be detected in the lock-on determination in step S18 (if lock-on cannot be performed), the lock-on flag is determined to be OFF in step S20 (No in step S20), and one frame is left as is. The process ends.

  If the input image is not an initial frame, “No” is determined in step S14, and the process proceeds to step S30. In step S30, it is determined whether or not the tracking flag is ON (step S30). If the tracking flag is OFF (No in step S30), the process proceeds to step S16.

  If it is determined in step S30 that the tracking flag is ON (Yes in step S30), the process proceeds to the eye tracking process in step S32. In the eye tracking process (step S32), the left and right eyes are individually tracked in order to monitor the open / closed state of the left and right eyes detected in the lock-on determination (step S18). Specifically, using the left and right eyes detected in the lock-on determination (step S18) as a model (initial model), pattern matching is performed around the position where each eye is detected, so that the position where the eye has moved is determined. Ask. If the score of the matching by the initial model (hereinafter sometimes referred to as similarity) falls below a predetermined reference value (first threshold), the latest updated model (“latest model”, Pattern matching is also performed by “the latest model” to detect the eye position.

  When the eye position cannot be confirmed, such as when the eye is hidden by an obstacle during tracking, the tracking is temporarily stopped, and when the eye can be confirmed again, the tracking is restored. Details of the eye tracking process will be described with reference to FIG. Further, processing related to tracking stop / return will be described with reference to FIGS.

  Following the eye tracking process (step S32 in FIG. 2), it is determined whether to open or close the eye being tracked in this eye tracking (step S34). Specifically, the same discriminator as the monocular detector used in the eye detection of the lock-on determination (step S18), that is, the discriminator (open eye detector) for detecting an open monocular obtained by machine learning. By checking the vicinity of the tracking position, it is determined whether to open or close the eye. Details will be described with reference to FIG.

  Open / close determination result information (D36) is obtained by the open / close determination process (step S34 in FIG. 2). After the opening / closing determination process (step S34), the process proceeds to step S40, and it is determined whether or not eye tracking is stopped. In the eye tracking process (step S32), when tracking is stopped for both eyes, the tracking flag is turned off (FIG. 5, step S238 described later). In this case, a Yes determination is made in step S40 of FIG. 2, and the lock-on flag is turned OFF (step S42).

  Next, the necessity of the re-lock on process is determined (step S44). If the lock-on flag is OFF, re-lock-on processing is performed. It should be noted that in a situation where tracking of only one eye is stopped and tracking of the other eye is being performed (situation waiting for return), it is determined that re-lock on is unnecessary.

  If it is determined in step S44 that relock-on is necessary (Yes in step S44), relock-on processing is performed (step S46). The re-lock-on process (step S46) is the same as the lock-on determination algorithm described in step S18.

  When the re-lock-on process (step S46) ends, the process for one frame ends. If it is determined in step S44 that the re-lock-on process is unnecessary, the process for one frame is terminated.

  The processing of FIG. 2 is repeated for each frame image acquired continuously (in time series) from the camera 12 described in FIG. 1 at a predetermined frame interval.

<Example 1 of lock-on determination processing flow>
FIG. 3 is a flowchart illustrating a first example regarding the flow of the lock-on determination process. When a frame image is input (step S102), binocular detection is performed by the binocular discriminator (step S104). This process is a process of collating a binocular discriminator against an input image and detecting a binocular candidate. The binocular discriminator is a detector that detects both eyes simultaneously, and specifies the approximate position of both eyes from the input image.

  Next, it is determined whether or not both eyes have been detected (step S106). If both eyes are detected in step S104, a Yes determination is made in step S106. If both eyes are not detected in step S104, step S106 is No. If NO in step S106, the process ends.

  If YES is determined in step S106, the process proceeds to step S108, and a monocular detection candidate position is set. In the monocular detection candidate position setting process (step S108), a position for collating the monocular discriminator is set for each binocular candidate detected by the binocular discriminator. In the present embodiment, the distance between the binocular center points is normalized to 32 pixels in the learning design of the binocular discriminator. Here, “pixel” represents a pixel of digital image data obtained from the camera 12. 32 pixels represents a distance of 32 pixels in a captured image of one frame.

  Therefore, it is determined that the center of the left and right eyes is located at the position of 16 pixels on both sides from the center of the detection position of the candidate for both eyes, and a predetermined range in the horizontal direction and the vertical direction in the image based on the center positions of the left and right eyes The monocular discriminator is applied within (within a relatively small range). In the monocular detection process by the monocular classifier (step S110), the monocular classifier is collated with the range set in step S108.

  As a result of the processing in step S110, it is determined whether or not two monoculars are detected (step S112). If two monoculars, that is, the right and left eyes are detected correctly, the two are detected. The target of tracking is locked on (step S114). When the left and right eyes can be identified as the tracking target, the lock on flag is turned ON.

  Then, both eyes that are locked on are set as an initial model (initial template) (step S116, corresponding to an “initial model generation step”). Eye tracking processing is performed using the set initial model (see FIG. 5).

  Note that if the determination in step S106 of FIG. 3 is No or the determination in step S112 is No, the processing for the frame is terminated.

  Also, during tracking of both eyes, which will be described later, the face size fluctuation is automatically detected from the distance between the centers of the left and right eyes, and a lock-on evaluation is performed locally to determine whether the same initial model can be used continuously. I do. The face size changes greatly, and the initial initial model cannot be applied (when the matching correlation is lower than the specified value), the lock-on process is performed again, and an appropriate initial model is set again.

<Example 2 of processing flow of lock-on determination>
FIG. 4 is a flowchart illustrating a second example regarding the specific processing flow of the lock-on determination. Instead of FIG. 3, the flowchart of FIG. 4 can be employed. In FIG. 4, after inputting a frame image (step S102), monocular detection is performed on the input image by a monocular discriminator (step S124). The monocular discriminator searches the input image and detects monocular candidates. By the monocular detection process (step S124), it is determined whether or not two or more monocular candidates are detected (step S126). If two or more monocular candidates are not detected, the determination is No in step S126, the process ends, and the process proceeds to the next frame image process.

  If two or more monocular candidates are detected by the process of monocular detection (step S124), a Yes determination is made in step S126, and the process proceeds to step S128.

  In step S128, symmetry is determined for two or more monocular candidates. Using the symmetry of the left and right eyes, it is determined whether or not the two monocular candidates are “binocular” in pairs (step S130). If a pair of two monocular candidates having left-right symmetry cannot be detected (No in step S130), the process ends and the process proceeds to the next frame image process.

  On the other hand, if the pair of eyes can be detected (Yes in step S130), the pair of eyes is set as a tracking target and locked on (step S134). When the left and right eyes can be locked on (when the eye can be identified as a tracking target), the lock-on flag is turned ON and both eyes of this pair are registered as the initial model (step S136, corresponding to the “initial model generation step”). Tracking processing is performed using the initial model thus set.

<Eye tracking process>
FIG. 5 is a flowchart showing the flow of eye tracking processing according to this embodiment. When the frame image is input (step S202), it is determined whether or not the eye is in the tracking state (step S204). Whether or not it is in a tracking state can be identified by a tracking flag (lock-on flag). If tracking is not in progress (No in step S204), lock-on determination (step S206) is performed to generate an initial model (step S208, corresponding to an “initial model generation step”). That is, immediately after the lock-on determination (step S206), the eye image portion detected by the lock-on determination is cut out as a template, stored in the memory as an initial model indicating the image characteristics of the eye, and held (step S208). The same processing as the processing illustrated in FIG. 3 or FIG. 4 is applied to the processing in steps S206 to S208.

  The initial model is also called an “initial template” and is used as the first reference template when tracking by template matching is started. This initial model is obtained by a lock-on process.

  On the other hand, if tracking has already been performed in the preceding frame (for example, the immediately preceding frame), the image portion of the eye tracked in the previous frame is cut out as a model and retained as an updated model (latest model) (See step S224 described later). In this example, the updated model is updated every frame, and the latest (most recent) model is always maintained during tracking. The update model is also called “latest model”, “latest model”, or “update template”.

  That is, an updated model is a model (template) obtained by using the image portion of the eye region cut out from the matching position of the immediately preceding frame. While maintaining the initial model generated by the lock-on process, the latest (most recent) update model is always generated, and the initial model and the latest update model are held.

  In step S204 of FIG. 5, it is determined that the tracking state is set (Yes in step S204), and the process proceeds to step S210. In step S210, a matching target area is cut out. This process is a process of cutting out a matching target area from the frame around the position of the eye detected (tracked) in the previous frame.

  Next, matching processing using an initial denplate (initial model) is performed (step S212, corresponding to “first matching processing step”). In the present embodiment, tracking is performed using both the initial template and the update template. Prior to the matching process using the update template, first, the template matching process using the initial template ("first template matching process") is performed. Equivalent) is performed (step S212). The similarity (corresponding to the “first score”) is calculated by matching the initial template against the matching target region. The similarity is calculated at each position while shifting the relative positional relationship between the matching target region and the initial template.

  The calculated similarity is compared with a first threshold (predetermined criterion value, for example, 50% similarity) (step S214, corresponding to “comparison process”). As a result of the matching, if a position having a similarity level equal to or higher than the first threshold (similarity level of 50% or more) is found, the result in Step S214 is Yes, and the position is determined as a tracking result (Step S216, “tracking position determination process”). ”).

  On the other hand, if a position having a similarity greater than or equal to the first threshold is not found in the matching by the initial template (step S212) (No in step S214), the template matching process ("second template" (Corresponding to “matching process”) (step S220, corresponding to “second matching process step”), and the position having the maximum similarity is set as the tracking result (step S222, corresponding to “tracking position determination step”). .

  When the tracking position is determined (steps S216 and S222), the eye region at the determined position is cut out as the latest model and recorded as the latest model (update template) (corresponding to step S224, “update model generation step”).

  As described above, in this embodiment, the matching process using the initial model is preferentially performed, and if the similarity of the initial model is equal to or higher than the specified threshold (first threshold), the result of the matching using the initial model is used as it is. . At this time, the computation of the matching process by the updated model is omitted, but the latest model is updated based on the matching result of the initial model.

  On the other hand, when the similarity of the initial model is lower than the first threshold, matching with the updated model is performed, and the matching result based on the updated model is adopted. In general, since the update model has a higher degree of matching than the initial model, the determination threshold value (second threshold value) when searching for a matching position by the update model is higher than the threshold value (first threshold value) of the initial model. Set to a high value.

  A series of steps including steps S212, S214, S216, S220, and S222 corresponds to the “object search step”.

  Next, a determination process for the tracking result is performed (step S226). Specifically, it includes a process of determining whether or not the initial model is appropriate (whether or not the same initial model can be used continuously) and a process of determining a tracking failure.

  First, in order to determine the suitability of the initial model, the distance between both eyes is obtained from the position of both eyes specified by eye tracking, and there is a condition (predetermined criterion value, It is determined whether or not “third threshold value” is exceeded (step S228). Note that the binocular distance increases as the face approaches the camera 12, and the binocular distance decreases as the face moves away from the camera 12. The third threshold value that defines the permissible range can include both a lower limit value and an upper limit value. Alternatively, by expressing the change amount of the binocular distance as an absolute value, the upper limit of the change amount (absolute value) can be set as the third threshold value.

  When the change amount of the binocular distance exceeds the third threshold value (Yes in Step S228), the perspective position of the driver's face with respect to the camera 12 may have moved, so the process proceeds to Step S230 and tries to re-lock on. .

  This re-lock-on process (step S230) is performed for the purpose of recreating the initial model (initial template). A new initial model is generated by the re-lock-on process (step S230), and the initial model is updated (step S232). The processing in steps S230 to S232 is processing as a measure of distance movement in which the driver's face moves away from the camera 12 or approaches the camera 12, and the initial model is recreated by determining the distance movement from the distance change of both eyes. Is. As the specific processing contents of steps S230 to S232, the same processing as the lock-on determination processing exemplified in FIGS. 3 and 4 is applied.

  If the change in the binocular distance is within the third threshold value in step S228, the process proceeds to step S236 to determine the validity of the tracking result. Specifically, the process shown in FIG. 6 is performed.

<Determining processing for poor tracking>
FIG. 6 is a flowchart illustrating an example of determination processing for performing tracking suspension or return processing from the stop based on the tracking result. FIG. 7 is an explanatory diagram showing transition relationships between three states that can occur during tracking in order to facilitate understanding of the flowchart of FIG.

  In the eye tracking process, both eyes are being tracked (denoted as “ST-2” in FIG. 7), and only one eye (one eye) is being tracked (tracking is stopped for the other eye) There are three possible states: a state in which the tracking is performed, indicated as “ST-1” in FIG. 7, and a state where tracking is stopped for both eyes (indicated as “ST-0” in FIG. 7). .

  The flow shown in FIG. 6 shows a determination processing procedure regarding stop and return of eye tracking. When data of a region of interest (ROI) to be tracked is input (step S302), it is first determined whether or not both eyes are being tracked (step S304). If both eyes are being tracked, the result of step S304 is Yes, and the process proceeds to the "Binocular simultaneous tracking stop determination" process in step S306.

  The binocular simultaneous tracking stop determination process (step S306) is a process for determining whether or not to stop the tracking of both eyes simultaneously while tracking both eyes. For example, when at least one of the following conditions 1 to 3 is satisfied, a process for simultaneously stopping the tracking of both eyes is performed.

  [Condition 1] Compared to the binocular distance immediately after lock-on (evaluated by the number of pixels between the center positions of both eyes), the current binocular distance is more than 1.25 times, and the correlation with the initial model (similarity) ) Is below 35%.

  [Condition 2] When the correlation of the eye with higher correlation (similarity) with the initial model is less than 20% and the correlation of the lower eye is 15% or less.

  [Condition 3] When the current binocular distance is narrower than the width of a single eye and the correlation of the eye with the higher correlation with the initial model is 35% or less.

  In addition, the specific numerical values “1.25 times”, “35%”, “20%”, and “15%” of each threshold value used as a criterion for the determination in the conditions 1 to 3 are merely examples, and are appropriately set within a target range. Can be set to a value. The binocular distance is an index for determining perspective variation (size change). That is, the determination condition is determined by combining the viewpoint of the size change and the viewpoint of the similarity with the initial model.

  After step S306, the process proceeds to step S308, where it is determined whether or not tracking has been stopped for both eyes. When tracking of both eyes is stopped simultaneously (Yes in step S308), the tracking flag is turned off (step S310). In this case, the process shifts to a re-lock on process (see steps S40 to S46 in FIG. 2 and ST-0 in FIG. 7).

  On the other hand, in the case of No determination in step S308 in FIG. 6, the process proceeds to the one-eye tracking stop determination process (step S312). If the binocular simultaneous tracking stop determination during the binocular simultaneous tracking does not occur (step S306), the single eye (monocular) tracking stop determination is performed (step S312). For example, when at least one of the following conditions 4 to 7 is satisfied, monocular tracking is stopped.

  [Condition 4] If the correlation of the eye with higher correlation (similarity) with the initial model exceeds 50% and the correlation of the lower eye with less than 15%, the eye with the lower correlation Stop tracking.

  [Condition 5] If the current binocular distance is 1.25 times or more away from the binocular distance immediately after Lock On, tracking of the eye with the lower correlation is stopped.

  [Condition 6] When the binocular distance becomes narrower than the lateral width of a single eye, tracking of the eye with the lower correlation is stopped.

  [Condition 7] When the correlation of the eye with a higher correlation with the initial model is higher than 50% and the correlation with the lower eye is lower than 50%, the current binocular distance is If the eye distance is less than a quarter of the eye distance, tracking of the eye with the lower correlation is stopped.

  In addition, the specific numerical values “50%”, “15%”, “1.25 times”, and “1/4” of the threshold values used as criteria for the determination in the conditions 4 to 7 are merely examples, and are within the target range. An appropriate value can be set.

  In the case of No determination in step S304, the process proceeds to step S320, and it is determined whether or not tracking is stopped for both eyes. If tracking is stopped for both eyes (Yes in step S320), the process proceeds to "Binocular simultaneous tracking return determination" in step S322. In the binocular simultaneous tracking return determination while the binocular tracking is stopped (step S322), matching of the initial model is performed for the tracking range held in the history for both eyes. If the initial model correlation exceeds 30% in this matching, tracking is resumed.

  In the case of No in step S320, that is, when one-eye (monocular) tracking is being stopped, the single-eye tracking return determination process in step S324 is performed. That is, when tracking is performed only for a single eye (tracking for one eye is stopped), a tracking return determination process is performed for the single eye for which tracking is stopped (step S324). For example, if the monocular being tracked is the left eye, the initial model is matched within the range of vertical 131 × horizontal 131 pixels centered on the position of the right 80 pixels from the center position of the tracking left eye. Also, if the single eye being tracked is the right eye, the initial model is matched within a range of 131 pixels in the vertical direction and 131 pixels in the horizontal direction from the center position of the right eye being tracked to the position of the left 80 pixels.

  If the initial model correlation exceeds 30% in this matching, tracking is resumed.

  Specific numerical values such as “80 pixels”, “131 pixels”, and “30%” of the threshold values indicating the above-described determination criteria are merely examples, and can be set to appropriate values within a target range.

  Subsequent to step S324, in step S326, it is determined whether or not the tracking is returned. If Yes in step S326, the process ends. If No in step S326, the process proceeds to the “one eye tracking stop determination” process in step S328. This step S328 is a process for determining whether or not to stop the tracking of the other eye during the monocular tracking stop. That is, when only a single eye is being tracked, processing for stopping the tracking of the single eye being tracked is performed. When the following condition 8 is satisfied, monocular tracking is stopped.

  [Condition 8] The correlation of the initial model is less than 25%, and it is determined as “closed eyes” in the opening / closing determination of the previous frame.

  Note that the specific numerical value “25%” of the threshold value that is the criterion for determination in Condition 8 is merely an example, and can be set to an appropriate value within the target range.

  The process described in FIG. 6 is used to determine the validity of the tracking result (step S236 in FIG. 5). If tracking of both eyes cannot be continued (No in step S236), the tracking state is turned off, that is, tracking is performed. The flag is turned off (step S238). On the other hand, if the tracking result is valid in step S236 (Yes in step S236), the tracking is continued and the processing of the next frame image is continued.

  As described above, as a method for determining tracking failure, the tracking failure of both eyes or one eye is detected using the matching position detected by matching using the initial model and the updated model and its score (similarity). Judgment can be made.

  Alternatively, tracking failure can be determined by using a score of opening / closing determination described later.

  Furthermore, it can be configured to include a step of storing information on tracking results of a plurality of frames as information. By using trajectory information grasped from the history of tracking result information of a plurality of frames, eye movement can be predicted (motion prediction), a bad tracking position can be determined, and tracking can be stopped. Here, “trajectory information” means a trajectory of a matching position (a movement trajectory of a position) between a plurality of continuous frames in time series. By storing the matching results of a plurality of frames as a history, a defective tracking position can be determined from the connection between the previous and next frames. For example, if the difference between the eye position at time t and the eye position at time t + 1 is greater than a prescribed value, it can be determined that tracking is poor.

  When it is determined that the tracking is poor, the tracking is stopped, the lock-on process is performed again, and the recovery of the tracking is attempted. The tracking failure determination process (tracking stop / return determination process) described with reference to FIG. 6 corresponds to a “tracking stop determination process”.

<About re-lock on>
The re-lock-on determination (step S46 in FIG. 1 and step S230 in FIG. 5) is performed when the correlation between the initial models falls below 80% in either the left or right eye, or when tracking of at least one eye is stopped To be done. This is because, after a certain amount of time has elapsed since eye tracking is started, a high correlation can be obtained even if matching processing is performed using the initial model due to distance fluctuation (distance movement) and illumination fluctuation between the driver and the camera 12. Therefore, the object is to update the initial model, or to supplement the tracking return processing (steps S322 and S324 in FIG. 6) and return the tracking as soon as possible when the tracking is stopped.

  The re-lock-on algorithm is substantially the same as the lock-on determination algorithm illustrated in FIGS. 3 and 4, but the position to collate the binocular discriminator is limited when tracking at least one of the eyes. Is done. That is, matching is performed as follows according to the three states ST-0, ST-1, and ST-2 shown in FIG.

  [1] When both eyes are being tracked (state ST-2 in FIG. 7), the binocular discriminator is collated in a relatively small range with reference to the midpoint of the center point of each eye.

  [2] When only one of the left and right (one eye) is being tracked (state ST-1 in FIG. 7), the tracking is stopped on the right side if the monocular being tracked is the left eye, and on the left side if the right eye is tracking. The binocular discriminator is collated in a range of 15 pixels wide and 10 pixels long based on the position of 40 pixels from the center of the single eye being tracked.

  [3] When both eyes are stopped (ST-0 in FIG. 7), the same binocular detection as in the first lock-on determination (step S18 in FIG. 1) is performed.

<Open / close judgment>
Next, the opening / closing determination process (step S34 in FIG. 2) will be described.

  FIG. 8 is a flowchart showing the flow of the open / close determination process. As pre-learning for initial setting required for open / close determination (step S402), a face angle indicating a face direction and a detection score (an evaluation value indicating similarity) when an open eye (monocular) discriminator is applied. Seeking a relationship with As a specific method, first, a plurality of analysis images for each face angle are prepared. The face angle, for example, represents the front-facing direction facing the camera 12 as a reference angle (0 degree), the right-handed rotation angle (horizontal angle) as a positive angle, and the left-handed rotation angle as a negative angle.

  Prepare face images for multiple people for each of various face orientation angles, apply a monocular open eye discriminator to each of these images, and from the obtained score distribution, the score for the front facing image, The relationship with the face angle is calculated by statistical processing.

  The open eye (monocular) discriminator is generated from a front-facing image. Therefore, although the score varies depending on the person, overall, the score of the front-facing image has the largest value, and the score value tends to decrease as the face angle increases. It becomes. By performing statistical processing using analysis images for a plurality of people, it is possible to grasp an average relationship.

  The processing of FIG. 8 is advanced on the premise that data by such prior learning is obtained in advance. When eye tracking is started, initial setting for opening / closing determination is first performed (step S402). This initial setting is a process of setting the relationship between the face angle and the score using the captured image of the driver who is the person to be judged.

  The accuracy of detection by the open eye discriminator varies depending on the person. For example, the detection score is large between a person wearing glasses and a person not wearing glasses. For this reason, if the statistical result obtained by prior learning is applied as it is, it may not always be possible to detect (determine) an appropriate opening. Therefore, the relationship between the face angle of the person to be judged and the detection score is set based on the statistical relation based on the prior learning based on the score of the person to be judged on the front image. That is, the process of step S402 is an algorithm that determines what threshold value is set for the person who is the person to be judged (the driver).

  Specifically, when the lock-on is performed, the history of the open / close determination result for each frame is held for 10 frames, and the average score value (“front The average detection score ") is calculated. And the relationship between a to-be-determined person's face angle and a detection score is determined from the statistical relationship obtained by prior learning.

An example is shown in FIG. The horizontal axis in FIG. 9 indicates the face angle magnitude (absolute value), and the vertical axis indicates the detection score value. In FIG. 9, g1 and g2 indicate the relationship between the face angle and detection score of different persons. For example, g1 is a graph of a person who is not wearing glasses, and g2 is a graph of a person who is wearing glasses.
The face angle is estimated from the eyes being tracked that are locked on by eye tracking (step S404 in FIG. 8) (step S406), and the relationship between the face angle and the detection score obtained in the initial setting (step S402) is estimated. A threshold value for opening / closing determination is set for each angle (step S408). That is, a threshold value represented by g1 or g2 in FIG. 9 is set. The threshold value is set to a smaller value as the face angle increases.

  Using the threshold value thus set (fourth threshold value), the open / closed state of the eye is determined in the subsequent score determination (step S414 in FIG. 8). The face angle estimation process will be described later with reference to FIG.

  Open / close determination is performed for each eye for each eye being tracked (step S410 in FIG. 8). That is, an open eye (monocular) discriminator is applied to determine whether to open or close the monocular, and an open eye is detected (step S412). An open eye (monocular) discriminator is applied to a predetermined image range determined by the tracking position specified in the eye tracking process, a score by open eye detection is calculated, and the threshold value (fourth threshold value) determined in step S408 The determination is made in comparison with (step S414).

  In the time axis integration process (step S416), the results of monocular opening / closing judgments for the latest several frames (for example, 2 to 3 frames) are accumulated, and the results of monocular opening / closing judgments for these recent several frames are integrated. The result is a monocular opening / closing judgment result. By such a time axis integration process, for example, it is possible to discard an instantaneous opening / closing change such as a blink, and a “blink” that is different from a doze can be ignored.

  For the left and right eyes, the processing of steps S410 to S416 is performed in parallel, and then the results of the monocular opening / closing determination for both the left and right are integrated (step S418), and the binocular opening / closing determination result as a comprehensive determination is output ( Step S420). Usually, a person often closes or opens the eyelids of both eyes at the same time, and it is difficult to close one eye and open the other. Therefore, in the present embodiment, when opening / closing is determined, the left and right monocular opening / closing determinations are comprehensively determined. That is, the final open / close determination is performed by combining the open / close determination results for each eye and integrating the open / close determination results for both eyes (binocular integration) (steps S418 to S420). Based on the output determination result, it is possible to determine whether the driver is falling asleep. Further, in addition to the dozing determination based on the above-described opening / closing determination, it is also possible to determine the side-view driving.

  Furthermore, in this embodiment, since the discriminator (open eye discriminator) constructed by machine learning of the image feature of the open eye is used for the open / close determination, the eye position is accurately determined in the open / close determination process. Specified. In the present embodiment, the tracking position is corrected based on the eye position thus detected. That is, an open / close determination is performed on the tracking position specified by the tracking process, and the eye position detected by the open / close determination is reflected in the tracking position. Such tracking position correction function enables tracking with higher accuracy.

<About face orientation (horizontal angle) estimation processing and side-arming determination>
FIG. 10 is a flowchart showing the flow of the face angle estimation process and the look-ahead determination process. As described above, after the eye to be tracked is locked on (step S502) and the setting of the relationship between the face angle of the person to be judged and the detection score is updated (step S504), the angle reference that is the reference for calculating the face angle Is set (step S506).

  FIGS. 11A and 11B are schematic diagrams for explaining a method of calculating a horizontal angle indicating the face direction (face angle). FIGS. 11A and 11B are schematic views of a person's head viewed from above with a cylinder approximation, where A represents the position of the left eye and C represents the position of the right eye. Arrow B indicates the direction of the face. FIG. 11A shows the front face orientation as a reference for calculating the face orientation horizontal angle, and FIG. 11B shows the face orientation rotated to the right in the horizontal plane from FIG. 11A.

  In the processing of the angle reference setting (step S506 in FIG. 10), when the lock is turned on, the linear distance between the left and right eyes (the linear distance D between A and C in FIG. 11A) is calculated from the captured image of the front. And record the value. Specifically, the distance between the center positions of the left eye and the right eye in the captured image (center distance between both eyes) is obtained in units of pixels, and the value is stored in the memory.

  Next, face angle estimation processing is performed from the tracking positions of both eyes specified by tracking (step S508 in FIG. 10) (step S510). In this angle estimation process (step S510), first, the center distance of both eyes (the linear distance Y between A and C in FIG. 11B) is calculated from the tracking position of both eyes.

  Then, the horizontal face orientation angle B is calculated by the following (Equation 1).

Horizontal face angle B = arcsin (Y / D) (Formula 1)
(Equation 1) is derived as follows.

  In FIG. 11, the center of the cylinder is E, the radius of the cylinder is r, and the center angle of both eyes (arc AC) is x [radians]. Further, in FIG. 11 (a), assuming that the reference line EF is orthogonal to the arrow B (vector EB), D is expressed by the following (Expression 2) from the relationship shown in FIG. 11 (a).

D = 2r · sin (x / 2) (Formula 2)
Note that “·” in the expression represents a multiplication operator (the same applies to Expression 3).

  On the other hand, in FIG. 11B, when the angle is measured counterclockwise from the reference line EF, the angle FEC is represented as angle C, the angle FEB is represented as angle B, and the angle FEA is represented as angle A. From the scientific relationship, the following (formula 3) to (formula 5) hold.

Y = r · cosC−r · cosA = 2r · sinB · sin (x / 2) (Formula 3)
C + X = A (Formula 4)
B = (A + C) / 2 (Formula 5)
From these relationships, the following (formula 6) is derived.

Y = DsinB (Formula 6)
Solving (Equation 6) with respect to angle B yields (Equation 1).

  If the angle B calculated in the angle estimation process (step S510 in FIG. 10) is compared with a predetermined threshold for looking aside, and the face orientation is equal to or greater than a predetermined angle and the state continues for a certain period of time, the driver It can be determined that the driver is looking aside (step S512).

  The result of the tracking process shown in step S508 of FIG. 10 is checked (result determination) for the tracking result (step S514). If the tracking result is OK, the process proceeds to the next frame (step S514). S516). On the other hand, if the tracking result is NG (NG in step S514), the process returns to step S502 and the lock-on process is performed again. When the camera 12 and subject rejection (perspective) change, the binocular linear distance D in the face image facing forward is changed, so that an appropriate re-lock-on is applied in order to minimize the error of D. By increasing the accuracy of D, the detection accuracy with respect to changes in face orientation is increased.

<Change of initial model corresponding to face angle>
In order to further improve the accuracy of template matching, it is preferable to adopt a configuration in which the initial model is updated when the face orientation changes greatly.

  FIG. 12 is a flowchart showing an initial model update method.

  As shown in FIG. 12, when both eyes are locked on by the lock-on determination process (step S602), an initial model when the lock-on is performed is recorded (step S604). This initial model is an initial front template when the driver's face is facing forward.

  At the start of tracking, tracking processing is performed using the initial model (initial front template) (step S608). Then, the orientation (face angle) of the currently tracked face is calculated by the method described in FIG. 11 (step S610). Then, based on the obtained face orientation angle, the initial model is subjected to deformation processing according to the angle, and a template corresponding to the face angle (referred to as “angle template”, expressed as “angle TP” in FIG. 12) is created. (Step S612).

  The angle template is created as follows (step S612).

  First, a peripheral area including the eyes of a person who is a three-dimensional subject (driver as a person to be judged) is approximated to a cylindrical shape, and the radius of the cylinder is determined according to the model size of the eye currently being tracked.

  Next, cylindrical rotation conversion is performed using the face orientation angle obtained in step S610 as a parameter.

  Finally, projective transformation is performed on the image after the rotation of the cylinder (orthographic projection on a plane), and an angle template corresponding to the current face orientation angle is generated. An angle template may be referred to as an “angle model” or a “rotation model”.

  FIG. 13 shows a conceptual diagram of a method for generating an angle template corresponding to a certain face orientation angle from the initial front template. As shown in the figure, an initial model corresponding to the face orientation angle is generated by performing cylindrical rotation conversion and projective conversion from the initial front model.

  The initial model thus obtained is stored as a new initial model, and the initial model is updated.

  In the tracking process for the next frame (step S618), the initial model (in this case, the angle template) updated in step S612 is applied.

  In this way, the face orientation angle is measured during tracking, and the initial model is adaptively rotated and deformed with respect to the measured angle, and corrected (updated) to an appropriate initial model. Improved accuracy can be achieved.

<Other examples of how to update the initial model>
Instead of the flow described in FIG. 12, when the initial front template is generated, a plurality of angle templates (rotation models) corresponding to a plurality of face orientation angles are created in advance, and the plurality of angle templates are generated. It is also preferable to store and save as initial template change candidates.

  For example, four types of angle templates corresponding to the face orientation angles of ± 30 degrees and ± 50 degrees are created using the front-front direction as a reference (rotation angle = 0 degree). In this case, a template candidate closest to the angle obtained in the angle estimation (step S610, corresponding to the “angle estimation step”) is selected and set as an initial model to be applied to the next frame (“store as a new initial model” ”).

  According to such a method, an appropriate initial model corresponding to the face orientation angle can be updated, and the accuracy of template matching can be improved. In addition, the calculation processing time can be shortened compared to the method described with reference to FIG.

  Note that the angle templates that are created in advance are not limited to the above four types, and those with appropriate angles can be created. Furthermore, not only the four types of angles illustrated above but also angle templates corresponding to a large number of angle positions may be created. In consideration of calculation time and detection accuracy, a rotation model with an appropriate number of types and angular positions may be created.

<Detailed Configuration of Image Processing Unit 20>
A configuration example of the image processing unit 20 that realizes the processing functions described with reference to FIGS. FIG. 14 is a block diagram illustrating a configuration example of the image processing unit 20. The image processing unit 20 includes an image input unit 202, a lock-on processing unit 204, a tracking processing unit 206, an open / close determination unit 208, a dozing determination unit 210, an aside look determination unit 212, and an initial model change unit 214. I have.

  The image input unit 202 is an image input interface unit for acquiring captured image data from the camera 12. Specifically, there may be aspects such as a communication interface unit, a data input terminal, and a media interface unit.

  The lock-on processing unit 204 includes a binocular discriminator 220, a monocular discriminator 224, an initial model generation unit 226, and an initial model storage unit 228. The lock-on processing unit 204 performs the lock-on process described with reference to FIGS. Further, when generating a plurality of angle templates in advance from the generated initial model (initial front template) at the time of generating the initial model, the lock-on processing unit 204 includes an angle template generating unit 230, an angle template storage unit 232, Is provided. The angle template generation unit 230 includes a cylindrical rotation conversion unit 236 and a projective conversion unit 238 for realizing processing similar to the conversion processing described with reference to FIG.

  The tracking processing unit 206 includes a first matching processing unit 250 that performs template matching processing using an initial model, a storage unit 252 that stores a first threshold value, a matching score using the initial model, and a first matching score. A score comparison unit 254 that compares the threshold values of 1 (corresponding to a “comparison unit”), a second matching processing unit 256 that performs template matching processing using the update model, and two matching processing units (250, 256) A tracking position determination unit 258 that determines a tracking position by using the result of the matching process, a tracking defect determination unit 260 that determines a tracking defect from the result of the matching process, and an updated model from the determined tracking position. An update model generation unit 262 to be generated, a storage unit 264 that stores the latest update model that has been generated, and an opening detection of the open / close determination unit 208 Using the result includes a track position correcting unit 266 for correcting the tracking position. The tracking processing unit 206 performs the eye tracking process described with reference to FIG. Further, the tracking failure determination unit 260 performs the eye tracking stop / return determination process described with reference to FIGS. 6 and 7.

  A combination of the first matching processing unit 250, the score comparison unit 254, the second matching processing unit 256, and the tracking position determination unit 258 corresponds to an “object search unit”.

  The opening / closing determination unit 208 includes an angle estimation unit 280 that calculates an angle of face orientation, a threshold determination unit 282 that determines a threshold (fourth threshold) for opening / closing determination according to the calculated (estimated) face angle, , An open eye (monocular) discriminator 284 that detects an open eye using the determined threshold, a time axis integration unit 286 that integrates discrimination results for the last several frames and determines opening and closing, and left and right monoculars A binocular integration unit 288 that performs determination by integrating the results of the single eye opening / closing determination performed for both eyes together. The opening / closing determination unit 208 performs the opening / closing determination process described with reference to FIG.

  Based on the opening / closing determination result output from the binocular integration unit 288, the dozing determination unit 210 determines whether the driver is in the dozing state.

  Further, based on the information on the angle estimated by the angle estimation unit 280, the sidewalk determination unit 212 determines whether or not the driver is in a sideward driving state.

  Further, the face angle information estimated by the angle estimation unit 280 is sent to the initial model change unit 214. The initial model changing unit 214 automatically selects an angle template most suitable for the face angle from the angle template storage unit 232 based on the face angle information, and changes the initial template.

  In addition, when it is detected by the dozing determination unit 210 or the look-aside determination unit 212 that a dozing state or a look-aside state is detected, a warning can be given to the driver by appropriate warning means (not shown). . As warning means, various forms such as warning means by voice (sound output means), warning means by display (warning display means), warning means for stimulating the driver such as vibrating the steering wheel (stimulation giving means) are possible. It may be an appropriate combination of these. Further, instead of or in combination with the warning means, based on the determination result of the dozing determination unit 210 or the aside look determination unit 212, driving assist control for avoiding danger (for example, automatic driving assistance such as automatic deceleration) (Control) is also possible.

  The function of each unit constituting the image processing unit 20 of the present embodiment is realized by hardware such as an integrated circuit, software (program) for operating a CPU (Central Processing Unit), or an appropriate combination thereof. be able to.

  Moreover, the function of each part which comprises the image process part 20 is realizable with a computer. That is, each process for realizing the image processing method described with reference to FIGS. 2 to 13 can be executed by a computer. A program for causing a computer to realize the image processing function described in the present embodiment may be installed in the computer in advance, or a magnetic disk, an optical disk, a magneto-optical disk, a memory card, or other computer storing the program. It is also possible to provide a readable medium (information storage medium). Further, instead of providing the program by storing the program in such a tangible storage medium, the program signal can be provided as a download service using a communication network such as the Internet.

  In addition, part or all of the image processing function described in this embodiment can be realized by cloud computing.

<How to set / update the initial model>
In the case of object tracking such as eye tracking, the conventional template matching technique has the following problems.

  (1) How to set the first template.

  (2) It is difficult to cope with changes in the size of the tracking target (for example, due to perspective variation) and angular changes (for example, due to horizontal rotation). For this reason, there is a possibility that tracking cannot be continued or a tracking position error occurs.

  In response to such a problem, according to the present embodiment, a lock-on process is performed using an object discriminator (in this example, a binocular discriminator 220, a monocular discriminator 224), and a reference template (initial model) is constructed. can do. Further, it is possible to re-lock on an object size change (perspective change) and angle (rotation) change to recreate a reference template (initial model). The change of the initial model at this time is performed, for example, if the matching correlation (score) with the initial template (front-facing front) is lower than a predetermined threshold under the condition that the object is in the forward range. Is possible.

  Furthermore, according to the present embodiment, the cylinder rotation transformation and the projective transformation are combined from the first reference template (for example, front-facing front) according to the standard angular position with respect to the change in the angle of the object (change in the rotation direction). By performing deformation processing, a reference template (rotation model) corresponding to the change in the angle of the object can be constructed.

  This makes it possible to set an initial model that is robust to changes in the size (perspective variation) and angle (rotation) of the object. In addition, the initial model can be adaptively modified / updated in response to a perspective (size) change or an angle change of the object to be tracked.

  When generating a front-facing initial model, a rotation model (an angle template) corresponding to a plurality of rotation angles in advance (for example, four types of ± 30 degrees and ± 50 degrees with respect to the front-facing direction) ) Can be created. By storing a plurality of angle template groups, it is possible to change the initial template by selecting the angle template having the closest angle based on the angle estimation of the object.

  That is, it is always determined whether or not the initial template currently used can cope with it, and when the matching correlation (score) becomes lower than the specified value, the initial template is changed (updated). Specifically, the re-lock-on process is performed to recreate the initial template.

  In addition to the aspect in which the initial template is re-created adaptively to the size change and the angle change, a plurality of templates corresponding to the size change and the angle change are generated from the front-facing reference initial template when the initial template is created. It is also possible to create in advance, select an appropriate template from these during tracking, and update the initial template actually used.

<Advantages of this embodiment>
(1) The conventional template matching technique cannot cope with the movement of the face (change in size) or the change in the face orientation angle, or the position error may accumulate. It can cope with the movement of the camera and the change of the face angle. In addition, since the initial model result is used preferentially and the updated model is used together, and the initial model is corrected (relocked on) in the case of poor tracking, the problem of accumulation of position errors is also solved.

  (2) When tracking both eyes of the driver, the driver usually looks at the front of the car in the driving direction, which is the basic posture (normal state) during driving, so the face when the driver is looking straight ahead The initial model is created from this face image facing forward, with the orientation as the reference “front”. When tracking using this initial model results in poor tracking, it is possible to correct the position error during tracking in order to recreate the initial model.

  (3) To prioritize the matching process using the initial model, and to perform tracking using the matching using the updated model (the latest model and the latest model) when the matching score based on the initial model falls below the first threshold value It is possible to cope with face direction variation and size variation (perspective variation).

  (4) Conventionally, the cumulative error that can occur by continuing to use the updated model is solved by using the initial model in the case of this embodiment, such as when the face orientation returns to the front. In this way, by using the multi-template (initial model and update model) giving priority to the initial model, the error accumulation can be corrected at a certain frequency, and the tracking accuracy is improved.

  (5) This embodiment includes an algorithm for accumulating tracking results of a plurality of frames and determining tracking stop from trajectory information (motion prediction). When the tracking is stopped, the mode is switched to the lock-on determination processing mode, and the initial model is recreated. In this way, it is updated to an appropriate initial model, and accurate tracking is possible.

  (6) Since the tracking position is corrected using the position information of the eyes detected by the open eye (monocular) discriminator used for open / close determination, the tracking accuracy can be further improved.

  (7) In the present embodiment, the driver uses a binocular discriminator 220 and a monocular discriminator 224 to perform lock-on and tracking and open / close determination using information on only the eyes of the facial parts. Judgment is possible even when wearing

  In this regard, the conventional face tracking method uses information on multiple facial parts such as eyes, mouth, and nose, so a part of the facial part is hidden when a person wears a mask. In this situation, the tracking object could not be detected. For this reason, there is a drawback that the process waits until the mask is removed and the determination is stopped. Further, in the conventional dozing technique (for example, Japanese Patent Application Laid-Open No. 2010-97379), the eyes are opened or closed by using general image features (for example, edges indicating outlines) extracted from the face image. There is a problem that the determination accuracy is low.

  According to this embodiment, since the open eye (monocular) discriminator constructed by machine learning is used, it is possible to make a judgment even when a mask is worn, and the judgment accuracy is improved as compared with the conventional method. .

<Modification 1>
In the above embodiment, during tracking, an update model is always created for each frame and is constantly updated to the latest update model. However, in implementing the invention, an update model is always created / updated for each frame. That is not required. The update model may be updated at appropriate frame intervals. Alternatively, one update model may be created from information on a plurality of frames.

<Modification 2>
In the above embodiment, one camera 12 is illustrated, but a system including a plurality of cameras may be used.

<Modification 3>
The application range of the present invention is not limited to the driver monitoring system, and can be applied to various other uses. It can be used as a general object tracking technique without any particular limitation on the use and tracking object.

<Modification 4>
As for the score representing similarity (matching degree, matching correlation degree) by template matching and the score by the object discriminator, those indicating that the degree of similarity is usually higher as the value is larger. However, depending on the definition of an index (evaluation value) representing the degree of similarity, the degree of similarity can be higher as the value is smaller. Accordingly, in determining similarity by comparing the score value and the threshold, attention is paid to the level of substantial similarity that each numerical value means.

  In the embodiment of the present invention described above, the configuration requirements can be appropriately changed, added, and deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by those having ordinary knowledge in the field within the technical idea of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Surveillance video system, 12 ... Camera, 16 ... Control apparatus, 20 ... Image processing part, 204 ... Lock-on processing part, 206 ... Tracking processing part, 214 ... Initial model change part, 220 ... Binocular discriminator, 222 ... Monocular discriminator, 226 ... initial model generation unit, 228 ... initial model storage unit, 230 ... angle template generation unit, 232 ... angle template storage unit, 250 ... first matching processing unit, 254 ... score comparison unit, 256 ... first 2 matching processing unit, 258 ... tracking position determination unit, 262 ... update model generation unit, 264 ... update model storage unit, 280 ... angle estimation unit, 282 ... threshold determination unit, 284 ... open discriminator

Claims (11)

An object tracking method for detecting a specific object from a captured image acquired in time series and tracking its position,
An initial model generation step of identifying an image portion of the object from the input captured image and generating an initial model indicating an image feature of the object;
An object search step of performing template matching processing using at least the initial model for a captured image input after generation of the initial model, and searching for the position of the object from the captured image;
An update model generation step of generating an update model from the image portion of the object at the tracking position specified by the object search step, and storing the latest update model;
Have
The object search step includes
A first matching processing step of performing a first template matching process using the initial model and calculating a first score indicating a similarity to the initial model;
A comparison step of comparing the first score obtained by the first matching processing step with a first threshold;
A second matching process step of performing a second template matching process using the latest update model stored, and calculating a second score indicating a similarity to the update model;
As a result of the comparison in the comparison step, when the first score shows high similarity equal to or higher than the first threshold, the matching position searched in the first matching processing step is adopted as the tracking position. On the other hand, an object including: a tracking position determining step that adopts a matching position searched by the second matching processing step as a tracking position when the first score shows a similarity lower than the first threshold value Tracking method.   After the initial model is generated, the first matching processing step is preferentially performed for each captured image that is sequentially input, and as a result of the comparison in the comparison step, the first score is the first threshold value. 2. The object tracking method according to claim 1, wherein the second matching processing step is omitted when high similarity equal to or higher than is shown.   The object tracking method according to claim 1, further comprising a tracking stop determination step of determining whether to stop or continue tracking based on a tracking result of the object search step.   The tracking stop determination step determines tracking stop based on at least one of the tracking position searched by the object search step, the first score, the second score, and the size of the object. The object tracking method according to claim 3.   The object tracking method according to claim 3 or 4, wherein when the tracking is stopped based on the determination in the tracking stop determination step, the initial model is recreated.   6. The object tracking method according to claim 1, further comprising a step of storing, as a history, tracking result information about a plurality of frames of captured images that are continuous in time series. An angle estimation step of calculating a horizontal angle indicating a horizontal rotation angle of the object from the tracking position determined by the tracking position determination step;
An initial model update step of changing the initial model according to the angle calculated in the angle estimation step;
The object tracking method according to claim 1, comprising: Based on the initial model generated in the initial model generation step, create a plurality of angle correspondence models corresponding to the plurality of rotation angles in advance,
One angle correspondence model is selected from the plurality of angle correspondence models based on the angle calculated in the angle estimation step, and the selected angle correspondence model is stored as a new initial model instead of the initial model. The object tracking method according to claim 7, further comprising a step.   9. The initial model generation step of generating the initial model by detecting an image portion of the object from the captured image using an object discriminator constructed by machine learning from a large number of images. The object tracking method according to claim 1. An object tracking device that detects a specific object from a captured image acquired in time series and tracks its position,
An initial model generation unit that identifies an image portion of the object from an input captured image and generates an initial model indicating an image feature of the object;
An object search unit for performing template matching processing using at least the initial model for a captured image input after generation of the initial model, and searching for the position of the object from the captured image;
An update model is generated from an image portion of the object at the tracking position specified by the object search unit, and an update model generation unit that stores the latest update model;
Have
The object search unit
A first matching processing unit that performs a first template matching process using the initial model and calculates a first score indicating a similarity to the initial model;
A comparison unit that compares the first score obtained by the first matching processing unit with a first threshold;
A second matching processing unit that performs a second template matching process using the latest stored update model, and calculates a second score indicating a similarity to the update model;
As a result of the comparison by the comparison unit, when the first score shows high similarity equal to or higher than the first threshold, the matching position searched by the first matching processing unit is adopted as the tracking position. On the other hand, when the first score indicates a similarity lower than the first threshold, a tracking position determination unit that employs the matching position searched by the second matching processing unit as a tracking position;
An object tracking device comprising:   The program for making a computer perform each process of the object tracking method of any one of Claim 1 to 9.