JP2008250999A - Object tracing method, object tracing device and object tracing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracing device excellent in a tracing accuracy in an object tracing processing to acquire a trajectory. <P>SOLUTION: The object tracing device for determining a trajectory of a tracing object in a moving image by repeating processing for acquiring position candidates for the tracing object in the current frame corresponding to the position candidates of the tracing object in the preceding frame acquires one or a plurality of position candidates of the current frame corresponding to one position candidate in the preceding frame. When there are a plurality of position candidates in the preceding frame, the object tracing device acquires position candidates of the current frame corresponding to the respective position candidates to thereby acquire a plurality of trajectory candidates of the tracing object, acquiring a plausible trajectory as a tracing result among the plurality of acquired trajectory candidates. The plausible trajectory is suitable to be selected on the basis of reliability in each position candidate. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for tracking a tracking object in a moving image.

  Conventionally, a technique for tracking (tracking) a human face or the like in a moving image has been used. In such tracking technology, a face detection is performed for each frame of a moving image and a tracking result is obtained by connecting the history, and one face-like region in the current frame is selected using the tracking result of the previous frame. Thus, there is a method for obtaining a tracking result from the initial frame to the current frame.

  The method of acquiring the position of the face in the current frame using the tracking result of the previous frame can limit the area to search for the face in the current frame, so it is faster and more reliable than the method that performs face detection every time Can track the face.

  However, there is a problem that once tracking is mistaken, tracking is erroneously performed in subsequent frames. For example, when the face being tracked intersects with an object resembling a face, if the object resembling the face is recognized as the tracking target, the object is tracked in subsequent frames. This will lead to malfunction. This malfunction is referred to as “nodding”.

  For example, as shown in FIG. 15, a case where a human face intersects with a vehicle wheel will be described as an example. In the example of FIG. 15, the face and the wheel overlap in FIG. 15 (d) while the face detected in FIG. 15 (a) is being tracked. Here, if the wheel is misrecognized as a face in FIG. 15E, the wheel is tracked in the frames after FIG. 15F. Therefore, as shown in FIG. 16, a route different from the original face is determined to be the locus of the face.

In order to solve such problems, feature points are detected from each frame of the moving image, a plurality of time-series associations of these feature points are listed, and the most reasonable association result among them is listed. Has been proposed as a tracking result (Patent Document 1). Here, the association of the feature points is determined in consideration of the similarity of the feature amounts at the corresponding feature points, whether the movement route (trajectory) is constant, or the like. According to such a method, temporally suitable feature point tracking can be realized without being affected by the interruption of the feature point correspondence that occurs in the middle of the moving image or the temporary erroneous correspondence.
JP 2004-5089 A

  However, the method described in Patent Document 1 performs feature point detection from the entire image region of each frame, and sufficiently uses the tracking result up to the previous frame to obtain the tracking result up to the current frame. Can't get to.

  It is an object of the present invention to improve tracking accuracy in an object tracking process for acquiring a track of a tracked object by acquiring a position of the object in the current frame by using a tracking result up to the previous frame. .

In order to achieve the above object, the object tracking method according to the present invention repeats the step of obtaining the position candidate of the tracking object in the current frame corresponding to the position candidate of the tracking object in the previous frame. In the object tracking method for determining the trajectory of the tracking object, one or more current frame position candidates corresponding to one position candidate in the previous frame are acquired, and when there are a plurality of position candidates in the previous frame, Acquiring a plurality of tracking object trajectory candidates by acquiring corresponding current frame position candidates for each position candidate, and determining the tracking object trajectory from the acquired plurality of trajectory candidates It is characterized by.

  In other words, the object tracking method according to the present invention is not limited to a single track of the tracked object, and allows a plurality of track candidates to be obtained by allowing branching in the middle. The trajectory of the tracking object is selected from among them.

  According to such a configuration, when the tracking object intersects with an object similar to the tracking object (hereinafter referred to as a similar object), the trajectory candidates are branched and tracking is performed for both. And, since the trajectory of the object is determined after tracking for both trajectory candidates, it is possible to prevent erroneous detection due to “pushing” even when the tracking object and the similar object intersect. it can.

  Further, the object tracking method according to the present invention calculates a reliability that is a probability that the tracking object exists in a predetermined area of the current frame corresponding to the position candidate of the previous frame, and the calculated reliability is a predetermined value. If it is equal to or greater than the threshold value, the area may be acquired as a position candidate of the tracking target in the current frame.

  Here, the reliability can be obtained as a detection score by template matching. In addition, the reliability may be calculated (corrected) based on the movement direction of the tracking object up to the previous frame by performing behavior prediction. Further, the reliability may be calculated using a color histogram or a matching technique. That is, the reliability may be calculated by any existing technique for obtaining a position candidate in the current frame based on a position candidate in the previous frame.

  According to such a configuration, it is possible to acquire the position candidate of the tracking object in the current frame at high speed and with high reliability. In addition, when the tracking target crosses a similar object and the trajectory candidate branches, it is later determined that the branch for the similar object was an error (there may be no area that seems to be a tracking target in the surrounding area). So that it is possible to abort the tracking for similarities.

  The object tracking method according to the present invention preferably stores the reliability of the position candidate of the tracking object in each frame, and determines the track of the tracking object based on the reliability of the locus candidate. is there.

  The determination of the trajectory may be performed by any specific method as long as it is performed based on the reliability related to the trajectory candidate. For example, the minimum value of the reliability (changes for each frame) in each trajectory candidate can be obtained, and the trajectory candidate having the largest minimum value can be determined as the trajectory of the tracking object. Further, an average value or cumulative value of reliability in each trajectory candidate can be obtained, and a trajectory candidate having a large value can be determined as the trajectory of the tracking target object. In addition, a trajectory candidate whose reliability in the trajectory candidate is once even below a predetermined threshold may be excluded or selected from trajectory candidates whose reliability in the trajectory candidate is higher than a predetermined threshold. good. In addition, the trajectory may be determined based on the maximum reliability value, variance value, standard deviation, and the like of the trajectory candidates. Moreover, you may utilize combining the method demonstrated here.

According to such a configuration, even when a trajectory candidate is generated due to the occurrence of “nodding” or the like, the reliability related to the trajectory candidate that tracks the similar object decreases with the passage of time. It is possible to select a correct locus from the candidates.

  In addition, in the object tracking method according to the present invention, when the position candidates of the current frame corresponding to the plurality of position candidates of the previous frame are the same, in the subsequent frames, one of the plurality of position candidates is selected. It is preferable to track the position candidates.

  According to such a configuration, when the trajectory candidates branch and then the branched trajectory candidates merge again, it is not necessary to track a plurality of position candidates, so the processing is speeded up.

  In addition, the object tracking method according to the present invention has an upper limit on the number of position candidates to be tracked, and when the position candidates acquired in the current frame exceed the upper limit, It is preferable to select a position candidate to be tracked in subsequent frames based on the reliability.

  According to such a configuration, it is possible to expect an increase in processing speed by narrowing down the number of trajectory candidates to be tracked.

  In the object tracking method according to the present invention, it is preferable that a tracking object is detected from a moving image, and the detected tracking object is tracked in subsequent frames.

  According to such a configuration, it is possible to track a tracking object that appears from the middle of a moving image. In addition, a plurality of tracking objects can be tracked simultaneously.

  In addition, the tracking target method according to the present invention, when the position of the detected tracking target is equal to the position candidate in the current frame of the tracking target that is already the tracking target, the detected tracking target is already detected. It is preferable that new tracking is not performed considering that the tracking target is a tracking target.

  According to such a configuration, when a method of detecting a tracking target object from a moving image and tracking the detected target object is adopted, even if an object being tracked is already detected, the tracking is repeated. Therefore, the processing can be made more efficient.

  Note that the object tracking method according to the present invention can track various objects as objects. For example, a human face can be adopted as the tracking target. In addition, other parts of the person (head, eyes, nose), the whole body, and the like can be employed as the tracking target. Further, the tracking object may be an arbitrary object.

  The present invention can also be understood as an object tracking method including at least a part of the above processing, or a program for realizing such a method. Moreover, this invention can also be grasped | ascertained as the target tracking apparatus which performs the said method. Each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

For example, an object tracking apparatus as one aspect of the present invention includes a moving image input unit that receives an input of moving images, and a position candidate of the tracking object in the current frame corresponding to the position candidate of the tracking object in the previous frame. The position candidate search means for acquiring one or a plurality of position candidates of the current frame for one position candidate in the previous frame, and corresponding to each position candidate when there are a plurality of position candidates in the previous frame Using the position candidate search means for acquiring the position candidate of the current frame, the trajectory candidate acquiring means for acquiring a plurality of trajectory candidates for the tracking object, and determining the trajectory of the tracking object from the acquired plurality of trajectory candidates And trajectory determination means.

  Further, in the object tracking device according to the present invention, the position candidate search means calculates a reliability that is a probability that the tracking object exists in a predetermined region of the current frame corresponding to the position candidate of the previous frame, When the calculated reliability is greater than or equal to a predetermined threshold, it is preferable to acquire the region as a position candidate of the tracking target in the current frame.

  Further, in the object tracking device according to the present invention, the trajectory candidate acquisition unit stores the reliability of the position candidate of the tracking target in each frame, and the trajectory determination unit includes the reliability related to the trajectory candidate. It is preferable to determine the track of the tracking object based on the degree.

  In the object tracking device according to the present invention, it is preferable that the tracking object is a human face.

  In addition, the object tracking program as one aspect of the present invention causes the information processing apparatus to repeat the step of acquiring the tracking object position candidate in the current frame corresponding to the tracking object position candidate in the previous frame. An object tracking program for determining a tracking object in a moving image, causing an information processing apparatus to acquire one or more current frame position candidates corresponding to one position candidate in a previous frame, and If there is a position candidate, a plurality of trajectory candidates for the tracking target object are obtained by acquiring a position candidate for the current frame corresponding to each position candidate, and the plurality of trajectory candidates obtained are The trajectory of the tracking object is determined.

  Further, in the object tracking program according to the present invention, the reliability that is the probability that the tracking target exists in a predetermined area of the current frame corresponding to the position candidate of the previous frame is calculated, and the calculated reliability is predetermined. It is preferable that the region is acquired as a position candidate of the tracking object in the current frame.

  Further, in the object tracking program according to the present invention, the reliability of the position candidate of the tracking object is stored in each frame, and the locus of the tracking object is determined based on the reliability of the locus candidate. Is preferred.

  In the object tracking program according to the present invention, the tracking object is preferably a human face.

  According to the present invention, it is possible to improve tracking accuracy in an object tracking process that determines the locus of a tracking object by acquiring the position of the object in the current frame using the tracking result up to the previous frame. It becomes.

  Exemplary embodiments of the present invention will be described in detail below with reference to the drawings. In the following description of the embodiment, a face tracking apparatus using a tracking target object as a human face will be described as an example, but the tracking target object may be other than a human face.

(First embodiment)
The face tracking apparatus according to the first embodiment performs tracking processing for all frames of an input moving image and then obtains a face movement locus.

<Configuration>
The face tracking device 1 according to the present embodiment is a normal computer (information) including a CPU (Central Processing Unit), a main storage device (RAM), an auxiliary storage device and the like connected via a bus in terms of hardware. Processing device). In this case, the face tracking device 1 is realized by executing the program on the CPU.

  FIG. 1 is a diagram illustrating an example of functional blocks of the face tracking device 1. The face tracking device 1 loads various programs (OS, applications, etc.) stored in the auxiliary storage device into the main storage device and is executed by the CPU, whereby the moving image input unit 2, the face detection unit 3, the trajectory candidate It functions as the acquisition unit 4, position information DB 5, trajectory information DB 6, trajectory determination unit 7, and image composition / output unit 8. Further, all or part of the face tracking device 1 may be configured as a dedicated chip.

  Next, each functional unit included in the face tracking device 1 will be described.

  The moving image input unit 2 functions as an interface for inputting moving image data to the face tracking device 1. The moving image input unit 2 may be configured using any existing technique for inputting moving image data to the face tracking device 1.

  For example, face image data may be input to the face tracking device 1 via a network (LAN or the Internet). Further, a moving image may be input to the face tracking device 1 from a digital video camera or a recording device (for example, a hard disk drive, a CD-ROM, a DVD-ROM, or various flash memories). Further, the face tracking device 1 may be included in an imaging device such as a digital video camera or various devices including an imaging device such as a digital video camera, and the captured moving image may be input to the face tracking device 1. .

  The face detection unit 3 detects a face from one frame of the input moving image. For example, the face detection unit 3 may be configured to detect a face by template matching using a reference template corresponding to the contour of the entire face. The face detection unit 3 may be configured to detect a face by template matching based on face components (eg, eyes, nose, ears). The face detection unit 3 may be configured to perform learning using a teacher signal using a neural network and detect whether a face is present in the search area. The face detection unit 3 may be realized by applying any other existing technology.

  The face detected by the face detection unit 3 is input to the trajectory candidate acquisition unit 4 and becomes a tracking target. Hereinafter, the face detected by the face detection unit 3 is referred to as a “basic face”.

  The locus candidate acquisition unit 4 includes a position candidate search unit 41. The position candidate search unit 41 acquires a face position candidate in the current frame corresponding to a face position candidate in the previous frame using a statistical pattern recognition method such as template matching. The position candidate search unit 41 sets a plurality of search areas in advance in the peripheral area of the face position candidate in the previous frame, and acquires a face position candidate in the current frame from among them by a statistical pattern recognition method.

  As shown in FIG. 2, a plurality of search areas As are determined in advance around the face position candidate Ac in the previous frame. Note that the number and arrangement of the search areas As may be arbitrary. The position candidate search unit 41 uses the image in the face position candidate Ac of the previous frame as the template image vector μ, and calculates the distance d between the image vector x and the template image vector μ in the search area of the current frame. Then, it is determined whether or not there is a face in the search area As. The distance d between the search image vector x and the template image vector μ is calculated according to the following equation.

  A search area in which the distance d between the search area image vector x and the template image vector μ of the candidate area in the previous frame is smaller than a predetermined threshold is set as a candidate area in the current frame.

  Note that the smaller the distance between the search image vector x and the template image vector μ, the higher the probability that the search image matches the template. That is, the distance d between the search image vector x and the template image vector μ can be regarded as the similarity between the search image and the template or the probability (reliability) that the search image and the template match.

  In the above description, the image of the previous frame is used as the template image vector μ. However, an image other than the immediately preceding frame (for example, an image when a face is first detected), an average image of a human face, or the like is used. You can use it.

  The position candidate search by the position candidate search unit 41 is configured to be able to acquire two or more position candidates for one position candidate in the previous frame. That is, when there are a plurality of search areas in which the distance between the search image vector x and the template image vector μ is less than or equal to the threshold (or the similarity or reliability is greater than or equal to the threshold), all the search areas are acquired as position information in the current frame To do. Further, when there are a plurality of position candidates in the previous frame, the position candidates corresponding to each are acquired from the current frame.

  The trajectory candidate acquisition unit 4 can acquire the trajectory of the face detected by the face detection unit 3 by applying the face tracking by the position candidate search unit 41 to each frame of the input moving image. it can. Here, since the position candidate search unit 41 may acquire a plurality of position candidates in the current frame, the locus candidate acquisition unit 4 acquires a plurality of locus candidates as a face locus.

  The position information DB 5 stores information related to face position candidates acquired by the position candidate search unit 41 (see, for example, FIG. 8B). In the position information DB 5, a position candidate ID 51, a frame number 52, an X coordinate 53, a Y coordinate 54, and a reliability 55 are stored as information on each position candidate. The position candidate ID 51 is an identifier for identifying a position candidate. The frame number 52 represents in which frame the position candidate is the position candidate. The X coordinate 53 and the Y coordinate 54 represent the position of the position candidate in the frame. The reliability 55 represents the probability that the position candidate is a face (the probability that a face exists in the position candidate region).

  The trajectory information DB 6 stores information on face trajectory candidates acquired by the trajectory candidate acquisition unit 4 (see, for example, FIG. 8C). The trajectory information DB 6 stores a trajectory ID 61, a basic face ID 62, and a path 63. The trajectory ID 61 is an identifier for identifying a trajectory candidate. The basic face ID 62 is an identifier for identifying the basic face that is the starting point of the trajectory. The path 63 indicates the position of the trajectory in each frame, and stores data in which the position candidate IDs of the position candidates stored in the position information DB 5 are connected.

  The trajectory determination unit 7 determines which trajectory is plausible from the plurality of trajectory candidates acquired by the trajectory candidate acquisition unit 4, and determines a single facial trajectory. Which trajectory is selected from among a plurality of trajectory candidates is determined based on the reliability at the time of each search.

For example, a trajectory candidate having the highest reliability average value during each search can be selected as a tracking result. That is, a route where the average value Ave _conf represented by the following equation is maximized is obtained with conf _i as the reliability at time i.

  For example, when two trajectory candidates are acquired as shown in FIG. 3 and the reliability at the time of each search takes the value shown in the figure, the average reliability is 645 for route A and 753 for route B. . Therefore, in the case of FIG. 3, the route B has higher reliability, so the route B is selected as the tracking result.

  In the above description, the trajectory candidate having the highest reliability average value is selected as the tracking result, but it may be selected by other methods. For example, based on the maximum and minimum values of reliability in the trajectory (that is, the trajectory with the largest maximum or minimum value) is selected as a tracking result, or selected by a statistical identification method using the moving direction, etc. You may do.

  The image composition / output unit 8 synthesizes a display that emphasizes the trajectory determined by the trajectory determination unit 7 with the input image, and outputs the result as a tracking result.

<Operation example>
Hereinafter, details of processing performed by each functional unit will be described according to an operation example of the face tracking device 1 according to the present embodiment.

  FIG. 4 is a flowchart showing an operation example of the face tracking device 1. The flowchart in FIG. 4 shows processing after a face is detected from a moving image.

[Initial processing for face detection]
When a face is detected from the moving image, the following initial processing is performed (S10). As shown in FIG. 5A, when a new face is detected from the moving image, information regarding the position of the face is stored in the position information DB 5. FIG. 5B is a diagram showing the contents of the position information DB 5 in which the newly detected face position and the like are stored. The reliability 55 representing the likelihood of a face is 1000 points (maximum value) because the face is detected by the face detection unit 3. Further, since a face is newly detected, a new record is added to the trajectory information DB 6 in order to obtain a trajectory for the face. In the path (trajectory) 63 of the newly added record, information on only one point of the detection position is stored.

  When the initial processing is completed, the next frame is set as a processing target (S11), and processing for acquiring a face position candidate in that frame is executed.

[Acquisition of face position candidates in the current frame]
Specifically, first, the position candidate search unit 41 acquires one position candidate in the previous frame (S12 to S13). The position candidate in the previous frame can be obtained by searching the position information DB 5 for the frame number 52 that is the previous frame.

  Next, the position candidate search unit 41 acquires a position candidate of the current frame corresponding to one position candidate in the previous frame acquired in S13 (S14). Specifically, the position candidate search unit 41 acquires face position candidates in the current frame as follows.

  First, as shown in FIG. 6, the search area in the current frame is determined in advance according to the position of the candidate face in the previous frame. In the example illustrated in FIG. 6, a total of nine search areas are defined, which are the same area 401 as the face candidate in the previous frame and eight areas 402 to 409 around the area 401. However, various methods can be adopted as a method for defining a search area.

  The position candidate search unit 41 calculates, for each of these search areas, a reliability that is a probability that a face exists in the area. Here, when calculating the reliability, correction by motion prediction that places importance on the moving direction to the previous frame may be performed. Further, correction may be performed in consideration of a face matching technique or a color histogram around the face. In addition, any existing technique for searching for a face from its surroundings in the current frame based on the position of the face in the previous frame may be applied.

  Then, an area where the calculated reliability is a predetermined threshold (for example, 500) or more is acquired as a face position candidate in the current frame. Since an area having a reliability greater than or equal to the threshold is set as a face position candidate, a plurality of face position candidates may be acquired for one face candidate in the previous frame, or one face position candidate may be obtained. It may not be acquired.

  Next, the position candidate search unit 41 determines how many position candidates have been acquired (S15). That is, it is determined how many search areas have a reliability greater than or equal to a threshold value. Here, the update processing of the position information DB 5 and the trajectory information DB 6 changes depending on how many position candidates are acquired.

[Update processing of position information DB / trajectory information DB]
1. When there is one position candidate When there is one position candidate acquired by the position candidate search unit 41, the process proceeds to S16, and new position candidate information is added to the position information DB 5. Then, a new position candidate is added to the path 63 of the trajectory information DB 6.

FIG. 7 shows an example in which the state shown in FIG. 5 is the previous frame, and only one face position candidate of the current frame for the face position candidate in FIG. 5 is acquired. As shown in FIG. 7B, information (position candidate ID: 2) related to the face position candidate in the current frame (f ₁ ) is added to the position information DB 5 (S16). Further, as shown in FIG. 7C, a position candidate (position candidate ID) newly added to the path candidate path 63 is added. Thus, the locus is extended in the locus information DB 6 (S17).

2. When there are two or more position candidates When there are two or more position candidates acquired by the position candidate search unit 41, the process proceeds to S <b> 18, and information on the plurality of position candidates is added to the position information DB 5. Then, the locus information DB 6 is updated (S19). Here, since a plurality of position candidates are acquired in the current frame, a branch occurs in the trajectory candidate. Therefore, after replicating the trajectory information in the trajectory information DB 6, new position candidates are added to the paths 63 of the respective trajectories.

FIG. 8 shows an example in which the state shown in FIG. 7 is the previous frame and two face positions of the current frame with respect to the face position of FIG. 7 are acquired. As shown in FIG. 8B, information (position candidate IDs: 3 and 4) regarding two face position candidates in the current frame (f ₂ ) is added to the position information DB 5. Further, as shown in FIG. 8C, the trajectory candidates are duplicated into two, and the newly added position candidates are added to the path 63 of each trajectory.

3. When there is no position candidate When there is no position candidate corresponding to the position candidate of the previous frame in the current frame, the trajectory is interrupted. In this case, since there is no corresponding position candidate in the current frame, the position information DB 5 is not updated. Since the trajectory ends, the trajectory information DB 6 is updated, a mark (flag) indicating the end of the trajectory is attached to the corresponding trajectory, and the fact that the trajectory is interrupted is stored (S20).

  When the above database update processing is completed, the process proceeds to S21, and it is determined whether the above processing is completed for all face candidates in the previous frame. If the above process has not been completed for all face candidates in the previous frame, the process proceeds to S13, and the above process is performed for the next face candidate. On the other hand, if the processing has been completed for all face candidates, the process proceeds to S22.

  In S22, it is determined whether one or more face candidates exist in the current frame. If a face candidate exists in the current frame, it is necessary to continue tracking the face, so that the process proceeds to S11 and the same processing as described above is executed for the next frame. If there is no face candidate in the current frame, no face is present in the moving image, so the process proceeds to S23, and the most likely face trajectory is determined.

[Trace determination process]
Hereinafter, details of the locus determination process in S23 will be described. Here, the case where three trajectory candidates are acquired by the trajectory candidate acquisition unit 4 as shown in FIG. 9A will be described as an example. 9A shows the trajectory of each trajectory candidate, FIG. 9B shows the contents of the position information DB 5, and FIG. 9C shows the contents of the trajectory information DB 6. As shown in FIG. 9A, two position candidates in the face frame f ₁ detected in the frame f ₀ are detected. Furthermore, in one locus, _two position candidates of the frame f ₂ corresponding to the position candidate detected in the frame f ₁ are detected. Therefore, for the detected face frame f _0, 3 single trajectory candidate is obtained.

  Here, the trajectory determination unit 7 calculates an average value of reliability (likeness of face) for each of a plurality of trajectories, and determines a trajectory having the highest reliability average value as a face trajectory. In the situation as shown in FIGS. 9A to 9C, the temporal change in the reliability for each locus is as shown in FIG. When the average value of reliability is obtained for each trajectory, the average value of the trajectory 1 becomes the largest when averaged over all frames, and the trajectory determination unit 7 determines the trajectory 1 as a face trajectory.

  Note that the determination method of the trajectory may be determined, for example, by selecting a trajectory having the largest reliability value. Alternatively, it may be determined by selecting a trajectory having the largest cumulative value of reliability. In addition, you may judge using statistics, such as a dispersion value and a standard deviation. In addition, the trajectory may be determined using not only the statistics for all frames but also the statistics for only the last few frames, and the trajectory may be determined using the reliability of only the last frame. Also good.

<Effect of this embodiment>
According to the face tracking device 1 according to the present embodiment, when a plurality of faces corresponding to the face of the previous frame are acquired in the current frame, the trajectory is branched and tracking is performed for each. Accordingly, when the face to be tracked intersects with an object similar to the face, it is possible to track both for the time being without determining the trajectory at that time, and to determine later which trajectory is correct. For example, in the situation shown in FIGS. 9A to 9C and FIG. 10, in the case where the face with the highest reliability is always tracked without allowing branching, the face is tracked for the locus 2 at point A (see FIG. 10). Will do. However, judging from the subsequent progress, it is found that the trajectory 2 is incorrect and the trajectory 1 is correct. According to the face tracking device according to the present embodiment, since a plurality of trajectories are tracked while allowing branching, the trajectory 1 can be obtained as a correct tracking result. That is, according to the face tracking device according to the present embodiment, face tracking can be performed with high reliability.

(Second Embodiment)
In the first embodiment, after the tracking process is completed for all the frames of the input moving image, a configuration is adopted in which the track is determined as one and used as a tracking result. In the present embodiment, an input of a moving image is accepted and one trajectory is determined for each frame. According to such a configuration, it is possible to output a face tracking result in real time while accepting input of a moving image.

  FIG. 11 is a flowchart showing the flow of face tracking processing in the present embodiment. The only difference from the face tracking process in the first embodiment is that the order of steps S22 and S23 is switched. As described above, in the present embodiment, since a trajectory candidate out of a plurality of trajectory candidates is determined and acquired as a tracking result in each frame, a tracking result can be obtained in real time.

  According to the present embodiment, the branch adopted as the tracking result may change with the passage of time. In such a case, the tracking result output at that time is changed. For example, when moving images such as those shown in FIGS. 9A to 9C and FIG. 10 are input, the trajectory 2 is output as a tracking result from time A to time B in FIG. And after passing the time B, the locus | trajectory 1 will be acquired as a tracking result. Therefore, the tracking result may be greatly changed by switching the locus candidates in this way.

  Specifically, a situation as shown in FIG. 15 will be described as an example. 12A to 12D correspond to FIGS. 15E to 15H. At the time of FIG. 12A, the average reliability calculated according to Equation 2 is 725 for route A (wheel) and 717 for face B (face), and route A is calculated with higher reliability. . Accordingly, in FIG. 12A, route A is displayed as a tracking result. However, at this time, the tracking of the route B is also continued.

  Next, at the time of FIG. 12B, when the average reliability is calculated according to Equation 2, the route B is calculated with higher reliability. Accordingly, in FIG. 12B, route B is displayed as a tracking result. In the subsequent FIGS. 12C and 12D, the reliability of the route B is calculated to be higher, so that the tracking result can be obtained correctly.

  According to the present embodiment, even when the tracking result is output in real time and the erroneous tracking occurs due to the dragging or the like (the trajectory 2 is selected at the time point A to B in FIG. 10), the erroneous tracking is also detected. It is possible to recover and obtain a correct tracking result (track 1 can be selected after time point B in FIG. 10). Thus, according to the face tracking device according to the present embodiment, face tracking can be performed with high reliability in real time.

(Third embodiment)
In the above-described embodiment, when a plurality of trajectory candidates once branched merge, processing for reducing the number of position candidates to be tracked is performed. In the above embodiment, when a branch occurs in the trajectory, the tracking process is performed for each trajectory. However, the face position in the next frame of the trajectory once branched may be the same region. In such a case, the plurality of locus candidates take the same locus in the subsequent frames. Therefore, it is possible to reduce the number of tracks to be tracked by merging without overlapping tracking. This can improve the processing efficiency.

  FIG. 13 is a diagram illustrating an example where two trajectories once branched merge. In FIG. 13, the trajectories branched at the position candidate 2 merge at the position candidate 7. In this way, when the position candidates of the next frame corresponding to the plurality of position candidates (5 and 6) are the same, only one position candidate is tracked in the subsequent frames.

  When the two trajectories merge at the position candidate 7, the trajectory information DB 6 stores a trajectory 1-2-3-5-7 and two trajectories 1-2-4-6-7. Regarding the handling of these two trajectories, the following two methods are conceivable.

  The first method is a method of simultaneously updating these trajectories when position candidates are acquired after the next frame. For example, when the position candidate 8 is acquired corresponding to the position candidate 7 in the next frame, the two loci are updated, and the loci of 1-2-3-5-7-8 and 1-2-4- The trajectory 6-7-8 is stored in the trajectory information DB 6. The second method is a method in which when these two trajectories merge, it is determined which trajectory candidate is the most likely trajectory, and trajectory candidates other than the plausible trajectory are deleted from the trajectory information DB 6. is there.

  According to the face tracking device according to the present embodiment, the processing speed can be increased.

(Fourth embodiment)
In this embodiment, an upper limit is set for the number of branches. That is, when the acquired position candidate exceeds the upper limit value, the position candidate exceeding the upper limit value is not tracked in subsequent frames. That is, the branch is treated as being interrupted in the next frame.

  When the number of position candidates exceeding the upper limit value is acquired in the current frame, position candidates to be tracked in the next frame and thereafter can be selected based on the reliability. For example, it is possible to employ a method of preferentially selecting a frame having a high reliability (reliability 55 of the position information DB 5) in the current frame acquired by the position candidate search unit 41. Also, for example, as in the locus determination process by the locus determination unit 7, which locus (position candidate) is based on the reliability related to the locus (average value, minimum value, maximum value, etc. of reliability in each frame of the locus). You may decide whether to select.

  According to the face tracking device according to the present embodiment, the upper limit value of the position candidate to be tracked is determined, and the track to be tracked is selected based on the reliability, so that the processing speed can be increased without causing a significant decrease in tracking accuracy. Can be realized.

(Fifth embodiment)
In the above description of the embodiment, the case where one face is tracked in the moving image has been described as an example. However, the face tracking device 1 can simultaneously track a plurality of faces from the moving image. That is, the face detection unit 3 detects a face from the input moving image every frame or every predetermined frame, and executes the face tracking process shown in FIG. 4 for each detected face. Thus, simultaneous tracking of multiple faces can be realized.

  Here, a problem may be that when the face detection unit 3 detects a face that is already being tracked, face tracking is performed twice for the same face. This problem can be avoided by the same method as in the fourth embodiment. Hereinafter, a method for avoiding the above problem will be described while explaining details of processing when a face is detected.

  FIG. 14 is a flowchart illustrating processing when a face is detected by the face detection unit 3. First, the face detection unit 3 detects a face from the input moving image (S101). When the face detection unit 3 detects a face (S102—YES), the face detection result and the face position candidate are merged. Specifically, it is determined whether the face position detected by the face detection unit 3 matches the position of the face candidate acquired by the position candidate search unit 41 (S103). ), The detected face is regarded as the same as the face being tracked, and no new tracking process is performed. When the face detected by the face detection unit 3 does not coincide with the face being tracked (S103-NO), the position information DB and the trajectory information DB are updated (S104). More specifically, the detected position of the face is stored in the position information DB 5 and an initialization process for adding one new basic face locus record to the locus information DB 6 is performed. Then, the tracking process (flow chart of FIG. 4) is started for the face (S105).

  According to the present embodiment, face tracking can be performed for a plurality of faces detected while detecting faces from a moving image, and it is possible to prevent a face being tracked from being tracked twice.

  The above embodiments are merely examples of the present invention. The scope of the present invention is not limited to the above embodiment, and various modifications can be made within the scope of the technical idea.

It is a figure which shows the functional block of a face tracking apparatus. It is a figure explaining the search area | region in a face detection process. It is a figure explaining the reliability at the time of a locus | trajectory candidate and each search. It is a flowchart which shows the flow of a face tracking process. It is a figure explaining update processing of position information DB and locus information DB when a new face is detected. It is the figure explaining the peripheral region for searching the face position of the present frame corresponding to the face position candidate of the previous frame. It is a figure explaining the update process of position information DB, locus | trajectory information, and locus | trajectory information DB when one face position candidate is acquired in the present frame with respect to the face position candidate of a front frame. It is a figure explaining the update processing of position information DB, locus information, and locus information DB when two face position candidates are acquired in the current frame with respect to face position candidates of the previous frame. It is a figure which shows each locus | trajectory candidate when a locus | trajectory branches and several locus | trajectory candidates are acquired. It is a figure which shows the content of position information DB corresponding to FIG. 9A. It is a figure which shows the content of locus | trajectory information DB corresponding to FIG. 9A. It is the figure which showed the example of the time change of the reliability which concerns on a some locus | trajectory candidate. It is a flowchart which shows the flow of a face tracking process in the case of outputting a tracking result in real time. It is a figure explaining reliability about each locus candidate at the time of each search in the case of outputting a tracking result in real time. It is a figure which shows the example which two branched locus | trajectory candidates merge. It is a figure which shows the flow of a process at the time of face detection in the case of tracking a some face simultaneously. It is a figure showing the face tracking process by a prior art. It is a figure showing the detection result of the face tracking process by a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Face tracking apparatus 2 Moving image input part 3 Face detection part 4 Trajectory candidate acquisition part 41 Position candidate search part 5 Position information DB
6 Trajectory information DB
7 Trajectory determination unit 8 Image composition / output unit

Claims (16)

An object tracking method for acquiring a track of a tracking object in a moving image by repeating a step of acquiring a position candidate of the tracking object in a current frame corresponding to a position candidate of the tracking object in a previous frame,
One or more current frame position candidates corresponding to one position candidate in the previous frame are acquired, and if there are a plurality of position candidates in the previous frame, the corresponding current frame position candidate is acquired for each position candidate. To obtain a plurality of trajectory candidates of the tracking object,
An object tracking method, comprising: determining a locus of the tracking object from a plurality of acquired locus candidates. A reliability that is a probability that the tracking target exists in a predetermined area of the current frame corresponding to the position candidate of the previous frame is calculated, and when the calculated reliability is equal to or greater than a predetermined threshold, the area is The object tracking method according to claim 1, wherein the object tracking method is obtained as a position candidate of a tracking object in a frame. Storing the reliability of the candidate position of the tracking object in each frame;
The object tracking method according to claim 2, wherein a locus of the tracking object is determined based on a reliability related to the locus candidate. When the position candidates of the current frame corresponding to a plurality of position candidates in the previous frame are the same, in the subsequent frames, one position candidate is tracked from among the plurality of position candidates. Item 4. The object tracking method according to any one of Items 1 to 3. There is an upper limit on the number of location candidates to track,
When a position candidate acquired in a current frame exceeds the upper limit value, a position candidate to be tracked in a subsequent frame is selected based on the reliability related to the acquired position candidate. 5. The object tracking method according to any one of 4 above. Detect tracking object from moving image,
The object tracking method according to claim 1, wherein the detected tracking object is tracked in subsequent frames. When the position of the detected tracking object is equal to the position candidate in the current frame of the tracking object that has already been tracked, the tracking object that has already been tracked is the detected tracking object The object tracking method according to claim 6, wherein new tracking is not performed.   The object tracking method according to claim 1, wherein the tracking object is a human face. Moving image input means for receiving moving image input;
A position candidate search means for acquiring a position candidate of a tracking object in a current frame corresponding to a position candidate of a tracking object in a previous frame, wherein the position candidate of the current frame is 1 for one position candidate in the previous frame If there are a plurality of position candidates in the previous frame, a plurality of tracking object trajectory candidates are acquired using position candidate search means for acquiring the corresponding current frame position candidate for each position candidate. Trajectory candidate acquisition means;
Trajectory determination means for determining a trajectory of the tracking target object from a plurality of acquired trajectory candidates;
An object tracking device characterized by comprising: The position candidate search means calculates a reliability that is a probability that the tracking target exists in a predetermined area of the current frame corresponding to the position candidate of the previous frame, and the calculated reliability is equal to or greater than a predetermined threshold. The object tracking device according to claim 9, wherein the region is acquired as a position candidate of the tracking object in the current frame. The trajectory candidate acquisition means stores the reliability in the position candidate of the tracking target in each frame,
The object tracking apparatus according to claim 10, wherein the trajectory determining unit determines a trajectory of the tracking target object based on the reliability related to the trajectory candidate.   The object tracking device according to claim 9, wherein the tracking object is a human face. An object for acquiring a track of a tracking object in a moving image by causing the information processing apparatus to repeat a step of acquiring a position candidate of the tracking object in the current frame corresponding to the position candidate of the tracking object in the previous frame. A tracking program,
In the information processing device,
One or more current frame position candidates corresponding to one position candidate in the previous frame are acquired, and when there are a plurality of position candidates in the previous frame, the corresponding current frame position candidate is acquired for each position candidate. To obtain a plurality of trajectory candidates of the tracking object,
An object tracking program for determining a locus of the tracking object from a plurality of acquired locus candidates. When the reliability that is the probability that the tracking target exists in the predetermined area of the current frame corresponding to the position candidate of the previous frame is calculated, and the calculated reliability is equal to or greater than a predetermined threshold, the area is The object tracking program according to claim 13, wherein the object tracking program is acquired as a position candidate of a tracking object in a frame. Storing the reliability of the candidate position of the tracking object in each frame;
The object tracking program according to claim 14, wherein a locus of the tracking object is determined based on a reliability related to the locus candidate.   The object tracking program according to claim 13, wherein the tracking object is a human face.

Images