JP5786879B2 - Subject tracking device, subject tracking method and program - Google Patents

Description

  The present invention relates to a subject tracking device, a subject tracking method, and a program.

2. Description of the Related Art Conventionally, motion capture is known in which a marker attached to a predetermined position of a subject is imaged and the motion of the subject is acquired (captured) by tracking the marker (see, for example, Patent Document 1).
This motion capture has been actively put into practical use in recent years. For example, in video creation using computer graphics (CG), a motion is acquired by motion capture, and the motion is given to the CG. Realizes video. For example, in the sports field, an action acquired by motion capture is taken into a computer and applied to an analysis of the action. In addition, the operation acquired by the motion capture is also used as the operation of the device.

Japanese Patent Laid-Open No. 7-55656

  By the way, tracking of a marker in image data generated by imaging is performed by using template matching. However, a system that is resistant to disturbance cannot be constructed only by specifying a marker position from the result of template matching. For example, when there are a plurality of good template matching results, it is difficult to determine which is best to select.

  Accordingly, an object of the present invention is to provide a subject tracking device, a subject tracking method, and a program capable of realizing tracking processing with high accuracy and excellent stability.

In order to solve the above problems, an object tracking device according to the present invention provides:
An acquisition means for acquiring a moving image of a subject; and a tracking means for tracking a specific part of the subject by processing using a particle filter in each of a plurality of frame images constituting the moving image acquired by the acquisition means; Regarding any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a corresponding region in a frame image before the one frame image, Search means for performing a route search on a predetermined graph by dynamic programming based on the difference between the plurality of particles of the specific part in the one frame image based on a search result by the search means Correction means for correcting likelihood, and a frame preceding the last frame image among the plurality of frame images by the correction means; After the likelihood of the plurality of particles at the specific part related to each of the frame images is corrected, the specification of each of the plurality of frame images is performed based on the tracking result of the specific part of the final frame image by the tracking unit. And a specifying means for specifying the tracking result of the part.

The subject tracking method according to the present invention includes:
A subject tracking method using a subject tracking device, wherein a specific part of a subject is obtained in each of an acquisition process for acquiring a moving image of a subject and a plurality of frame images constituting the moving image acquired by the acquisition process. A tracking process to be tracked by a process using a particle filter, and any one of the plurality of frame images, and any one of the plurality of particles generated by the tracking process for each specific part One of the above based on a search process for performing a route search on a predetermined graph by dynamic programming based on the degree of difference from the corresponding region in the frame image before the frame image, and a search result by the search process A correction process for correcting the likelihood of the plurality of particles in the specific part in one frame image; Based on the tracking result of the specific part of the final frame image by the tracking process after correcting the likelihood of the plurality of particles of the specific part of each frame image before the final frame image of the frame image Specific processing for specifying a tracking result of the specific part of each of the plurality of frame images.

The program according to the present invention is
The computer of the subject tracking device is configured to acquire a moving image of the subject by processing using a particle filter in each of a plurality of frame images constituting the moving image acquired by the acquiring unit. A tracking unit for tracking, and for any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a frame image before the one of the frame images Search means for performing a route search on a predetermined graph by dynamic programming based on the degree of difference from the corresponding region in the above, based on the search result by the search means, the specific part of the frame image Correction means for correcting the likelihood of a plurality of particles, and a maximum of the plurality of frame images by the correction means; After correcting the likelihood of the plurality of particles of the specific part related to each of the frame images before the frame image, the plurality of frames based on the tracking result of the specific part of the final frame image by the tracking unit It is characterized by functioning as a specifying means for specifying the tracking result of the specific part of each image.

  According to the present invention, tracking processing with high accuracy and excellent stability can be realized.

It is a block diagram which shows schematic structure of the to-be-photographed object tracking apparatus of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to a subject tracking process by the subject tracking device of FIG. 1. 3 is a flowchart showing a continuation of the subject tracking process of FIG. 2. 3 is a flowchart illustrating an example of an operation related to a particle likelihood correction process in the subject tracking process of FIG. 2. It is a figure for demonstrating the subject tracking process of FIG. It is a figure for demonstrating the subject tracking process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram showing a schematic configuration of a subject tracking apparatus 100 according to an embodiment to which the present invention is applied.

  The subject tracking device 100 is configured by a computer such as a workstation, for example, and as shown in FIG. 1, a central control unit 1, a memory 2, a storage unit 3, an operation input unit 4, a tracking processing unit 5, and the like. , A display unit 6, a display control unit 7, and an external communication unit 8.

  The central control unit 1 controls each unit of the subject tracking device 100. Specifically, the central control unit 1 includes a CPU (Central Processing Unit; not shown), and performs various control operations according to various processing programs (not shown) for the subject tracking apparatus 100.

  The memory 2 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores data or the like processed by the central control unit 1, the tracking processing unit 5, and the like. The memory 2 temporarily stores the moving image data D transmitted from the external device 200 to the subject tracking device 100 and received by the external communication unit 8.

  The storage unit 3 is composed of, for example, a nonvolatile memory (flash memory) or the like, and stores various programs and data (not shown) necessary for the operation of the central control unit 1.

  The operation input unit 4 includes, for example, a data input key for inputting numerical values, characters, and the like, a keyboard, a mouse, and the like configured by up / down / left / right movement keys and various function keys for performing data selection, feed operation, The operation unit (not shown) is provided, and a predetermined operation signal is output to the central control unit 1 in accordance with the operation of these operation units.

The tracking processing unit 5 includes a moving image acquisition unit 5a, a template generation unit 5b, a tracking unit 5c, a particle selection unit 5d, a route search unit 5e, a likelihood correction unit 5f, and a tracking result identification unit 5g. It has.
In addition, although each part of the tracking process part 5 is comprised from the predetermined logic circuit, for example, the said structure is an example and is not restricted to this.

The moving image acquisition unit 5a acquires a moving image of the subject S.
That is, the moving image acquisition unit (first acquisition means) 5a acquires moving image data D of the subject S including a plurality of frame images F,. Specifically, the moving image acquisition unit 5 a acquires the moving image data D of the subject S generated by the external device 200 from the memory 2.
Here, the moving image data D stores a plurality of frame images F,... In which the subject S performs a predetermined operation (for example, a golf swing).

Examples of the external device 200 include an imaging device that captures a moving image of the subject S and a computer that acquires the moving image data D of the subject S from the imaging device. Any configuration can be used as long as it can be transmitted to the subject tracking apparatus 100, and detailed description thereof will be omitted. Further, the external device 200 may be configured to sequentially transmit the frame images F constituting the moving image data D generated by the imaging of the subject S, or after the imaging of the subject S is finished, the moving image data D is collected. The structure which transmits may be sufficient.
The moving image acquisition unit 5a may acquire the moving image acquisition unit 5a via a recording medium (not shown) that is detachable from the subject tracking device 100 main body.

The template generation unit 5b generates a template Pt for the specific part P of the subject S.
Specifically, the user's operation in the first frame image F1 (see FIG. 5A) among the plurality of frame images F,... Constituting the moving image data D acquired by the moving image acquisition unit 5a. Initial positions (for example, joint positions such as the left wrist, left elbow, and left shoulder) relating to a specific portion (for example, the left arm) P to be tracked based on a predetermined operation of the input unit 4; black circles in FIG. Is designated), the template generation unit 5b generates the color and shape (thickness) of the specific part P as the template Pt from the first frame image F1. For example, the template generation unit 5b specifies the color of the constituent parts (for example, the upper arm, the forearm, etc.) constituting the specific part P from the initial position in the first frame image F1, and then determines the thickness of the forearm and upper arm from the specified color. Is specified (see FIG. 5B).
That is, if the subject S is instructed in advance to wear a forearm and upper arm that have as much a single color as possible (for example, fine patterns can be removed by filtering), a template is generated. The unit 5b can specify the color of the upper arm from between the left shoulder and the left elbow specified as the initial position, and can specify the color of the forearm from between the left elbow and the left wrist. Then, the template generation unit 5b prepares and compares a plurality of shapes such as upper arms and forearms having appropriate widths, and adopts a shape such as the upper arms and forearms that has been determined to have the highest degree of matching.

The tracking unit 5c tracks the specific part P of the subject S in each of the plurality of frame images F,.
That is, the tracking unit (tracking unit) 5c uses a particle filter for the specific part P of the subject S in each of the plurality of frame images F,... Constituting the moving image data D acquired by the moving image acquiring unit 5a. Track by processing
Here, the particle filter is useful in time series analysis of a signal to which non-Gaussian noise that cannot be handled by the Kalman filter is added, for example. In addition, the particle filter can express a target probability distribution by introducing a finite number of particles Pp as tracking candidates, and can perform time series estimation and prediction using the target probability distribution. Detailed description is omitted here.

  Specifically, the tracking unit 5c includes a particle generation unit c1 and a dissimilarity calculation unit c2.

Among the plurality of frame images F,... Of the moving image data D, the particle generation unit c1 sets the initial position (for example, the left wrist) for each specific part P of the subject S to be tracked specified in the first frame image F1. , Left elbow, left shoulder, etc.), random noise is added to the position coordinates to generate particles Pp in a plurality of states (for example, N; N is a natural number of 2 or more) within the framework of the particle filter (FIG. 5 ( c)).
Here, the particle Pp represents one state related to the specific part P such as the left arm, the right arm, the left leg, the right leg, the trunk, and the head. Specifically, the particle Pp includes, for example, the position coordinates of the initial position (for example, the left shoulder, the left elbow, the left wrist, etc.) related to the designation of the specific part P and the constituent parts (for example, the upper arm, Information on the width of the forearm, etc.). FIG. 5C schematically shows an example of the left arm particle Pp.

The difference calculation unit c2 calculates, for each specific part P, the degree of difference between each of the plurality of particles Pp,... And the template Pt of the specific part P to be processed.
That is, the difference calculation unit c2 differs between the plurality of particles Pp generated by the particle generation unit c1 for each specific part P and the template Pt of the specific part P for one frame image F to be processed. Calculate the degree DIFF. Specifically, the difference calculation unit c2 calculates the difference DIFF using, for example, a difference square sum SSD (Sum of Squared Differences).
For example, the dissimilarity calculation unit c2 extracts the upper arm state and the forearm state from any one of the plurality of particles Pp for the left arm,... (For example, the state of the left arm model), and generates a template. Compared with the template Pt generated by the unit 5b, the upper arm and forearm dissimilarities UD and LD are calculated. Then, the dissimilarity calculating unit c2 performs a predetermined calculation based on the calculated dissimilarities UD and LD of the upper arm and the forearm (for example, adding the dissimilarities UD and LD), and the dissimilarity of the particle Pp Calculated as DIFF. The difference calculation unit c2 performs the same process on the other particles Pp, and calculates each difference DIFF.

  Further, the tracking unit 5c is based on a plurality of (for example, N) particles Pp related to one frame image F whose likelihood is corrected by the likelihood correcting unit 5f, and is later than the one frame image F. The specific part P of the frame image F is tracked. Specifically, when the processing target is advanced to the next frame image F, the particle generation unit c1 adds random noise to the plurality of particles Pp,. The unit 5c tracks the specific part P in the next frame image F using the plurality of particles Pp,.

The particle selecting unit 5d selects a predetermined number of particles Pp among the plurality of particles Pp,.
That is, the particle selection unit (selection unit) 5d, for one frame image F to be processed, among the plurality of particles Pp generated by the particle generation unit c1 for each specific part P,. A predetermined number of particles Pp are selected based on the degree of difference from the template Pt. Specifically, the particle selection unit 5d uses the difference degree DIFF calculated by the difference degree calculation unit c2 for a plurality of (for example, N) particles Pp,... As a reference in order from the smallest difference degree DIFF. A number of particles Pp (for example, K; K is a natural number smaller than N) is selected.
FIG. 6A and the like illustrate four particles Pp (nodes 11 to 14, 21 to 24, 31 to 34, etc.) selected by the particle selection unit 5d. It is not limited, and the selected number can be arbitrarily changed as appropriate.

The route search unit 5e performs route search on a predetermined graph by dynamic programming, and calculates the arrival cost of each route.
That is, the route search unit (search unit) 5e, for one frame image F to be processed, has a plurality of particles Pp generated by the tracking unit 5c for each specific part P, and a plurality of frame images F,. A route search on a predetermined graph (for example, a weighted directed graph G) by dynamic programming based on the difference from the corresponding region in all the frame images F before the one frame image F constituting I do.
Specifically, for one frame image F to be processed, the route search unit 5e sequentially adds a plurality of particles Pp, ... generated by the particle generation unit c1 for each specific part P to the weighted directed graph G. The route search is performed on the weighted directed graph G by dynamic programming. More specifically, for each frame image F, the route search unit 5e sequentially adds K particles Pp selected by the particle selection unit 5d for each specific part P to the weighted directed graph G, and A route search is performed on the basis of the degree of difference between Pp and the corresponding area in the frame image F before the frame image F to be processed. Further, for each frame image F, the route search unit 5e determines the difference between each of the plurality of particles Pp,... Generated by the particle generation unit c1 for each specific part P and the template Pt of the specific part P, that is, A route search is performed on a predetermined graph by dynamic programming based on the difference DIFF of each particle Pp calculated by the difference calculation unit c2.
For example, the difference degree DIFF of each particle Pp calculated by the difference degree calculation unit c2 for the K (for example, four, etc.) particles Pp selected by the particle selection unit 5d for each specific part P is a weighted directed graph. It becomes a node (nodes 11-14, 21-24, 31-34, etc.) for each frame image F of G (for example, frame images F1, F2, F3, etc.) (see FIG. 6 (a), etc.). When the processing object moves to the next frame image F, the difference DIFF of K particles Pp is newly added to the node of the weighted directed graph G for each specific part P of the frame image F (FIG. 6A ) And FIG. 6B). The direction of the edge connecting the nodes is the direction toward the newly added particle Pp, and the weight (path cost E) changes according to the state change amount between the two adjacent frame images F and F.
For example, the route search unit 5e, according to the following formula, represents representative points that constitute a specific part P of two adjacent frame images F, F (for example, when the specific part P is the left arm, the left shoulder, the left elbow, the left wrist, etc.) A value obtained by multiplying the Euclidean distance (inter-coordinate distance) D in the predetermined color space of the position coordinates by the adjustment parameter α and the difference degree DIFF of the particles Pp is set as the edge path cost E.
Route cost E = Dissimilarity DIFF + α x D
In FIGS. 6A to 6C, nodes are represented by white circles and edges are represented by solid lines. In addition, it is assumed that the nodes are arranged from top to bottom in descending order of likelihood of the particles Pp.

Further, the route search unit 5e configures a plurality of frame images F,... For each of the plurality of particles Pp,... Generated by the particle generation unit c1 for each specific part P in the frame image F to be processed. The arrival cost of a route from a predetermined reference frame image (for example, the first frame image F1) to the processing target frame image F is calculated by a dynamic programming method. Specifically, for example, for the K (for example, four) particles Pp selected by the particle selection unit 5d, the path search unit 5e starts processing from the first frame image F1 to the processing target frame image F (for example, The arrival cost of the route to the third frame image F3 or the like is calculated (see FIG. 6B).
In FIG. 6B, the shortest path with the minimum arrival cost from the first frame image F1 to the third frame image F3 to be processed is represented by a bold line, and the arrival cost to each of the nodes 31 to 34 is shown. Represented by numbers in parentheses.
In addition, since the route search by dynamic programming is a well-known technique, detailed description is abbreviate | omitted here.

The likelihood correcting unit 5f corrects the likelihood of a predetermined number of particles Pp.
That is, the likelihood correcting unit (correcting unit) 5f corrects the likelihood of each particle Pp of the specific part P in the one frame image F to be processed based on the search result by the route searching unit 5e. More specifically, the likelihood correcting unit 5f calculates each particle Pp based on the arrival cost of the path from the first frame image F1 for each particle Pp calculated by the path searching unit 5e to the frame image F to be processed. Correct the likelihood. For example, the likelihood correcting unit 5f sets the ratio of the reciprocal of the arrival cost of the path for each particle Pp calculated by the path searching unit 5e as the likelihood of the particle Pp. Then, the likelihood correction unit 5f normalizes the inverse of the path cost of each particle Pp by N. For example, if the path cost of the particles Pp of the nodes 31 to 34 is 120: 340: 100: 800, the likelihood of the ratio of 1/120: 1/340: 1/100: 1/800 = 0.37: 0.13: 0.44: 0.06 Degree. Then, the likelihood correcting unit 5f multiplies the likelihood of each of the nodes 31 to 34 by N to set a total of N particles Pp.

The tracking result specifying unit 5g specifies the tracking result of the specific part P in the plurality of frame images F,.
That is, the tracking result specifying unit 5g includes a plurality of particles Pp,... Of the specific part P related to each of the frame images F before the final frame image Fn of the plurality of frame images F,. , The tracking result of each specific part P of the plurality of frame images F,... Is specified based on the tracking result of the specific part Pp of the final frame image Fn by the tracking unit 5c. Specifically, the tracking result identifying unit 5g reaches the path arrival cost for each particle Pp from the first frame image F1 to the last frame image Fn among the plurality of frame images F calculated by the route searching unit 5e. , The tracking result of each specific part P of the plurality of frame images F,.
For example, after calculating the arrival cost of the plurality of routes for each particle Pp for the final frame image Fn by the route search unit 5e, the tracking result specifying unit 5g has the lowest arrival cost among the plurality of routes for which the arrival cost is calculated. Is identified (see FIG. 6C). In other words, the path search unit 5e uses the difference DIFF of each particle Pp (node) after correction and the path cost E of each edge for the frame image Fn-1 immediately before the first frame image F1 to the last frame image Fn. Based on the final frame image Fn, the arrival cost of each path for each particle Pp is calculated. Then, the tracking result specifying unit 5g specifies the shortest path with the lowest arrival cost, and specifies each particle Pp of each frame image F related to the shortest path as the tracking result of the specific part P.
In FIG. 6C, the shortest path with the lowest arrival cost in all the frame images F1 to Fn is indicated by a bold line.

  The display unit 6 includes a display such as an LCD (Liquid Crystal Display) and a CRT (Cathode Ray Tube), and displays various types of information on the display screen under the control of the display control unit 7.

The display control unit 7 performs control to generate display data and display it on the display screen of the display unit 6.
Specifically, the display control unit 7 includes a video card (not shown) including, for example, a GPU (Graphics Processing Unit), a VRAM (Video Random Access Memory), and the like. Then, in accordance with the display instruction from the central control unit 1, the display control unit 7 generates moving image display data based on the moving image data D acquired by the moving image acquisition unit 5a by drawing processing using a video card, Output to the display unit 6.

  The external communication unit 8 is connected to the external device 200 so as to be able to transmit and receive information via a predetermined communication line (for example, a LAN (Local Area Network) or the like). Specifically, the external communication unit 8, for example, omits a predetermined communication cable (for example, a LAN cable) attached to a terminal (for example, a LAN terminal or the like) for connection with the external device 200 although not illustrated. Send and receive data via For example, the external communication unit 8 receives the moving image data D transmitted from the external device 200. The moving image data D received by the external communication unit 8 is transferred to the memory 2 and temporarily stored.

<Subject tracking process>
Next, subject tracking processing by the subject tracking device 100 will be described with reference to FIGS.
2 and 3 are flowcharts showing an example of an operation related to the subject tracking process.
In the following description, it is assumed that the moving image data D is acquired by the external device 200 and then temporarily stored in the memory 2 via the external communication unit 8.

As shown in FIG. 3, first, the moving image acquisition unit 5a reads out and acquires the moving image data D of the subject S from the memory 2 (step S1).
Next, in the first frame image F1 (see FIG. 5A) of the moving image data D acquired by the moving image acquisition unit 5a, the identification of the tracking target based on a predetermined operation of the operation input unit 4 by the user. When an initial position (for example, left wrist, left elbow, left shoulder, etc.) related to the part P is specified (step S2), the template generation unit 5b configures the specific part P from the initial position in the first frame image F1. The color of the part (for example, the upper arm, the forearm) and the thickness of the forearm and the upper arm are specified, and the color and shape (thickness) of the specific part P are generated as the template Pt (step S3).

  Subsequently, the particle generation unit c1 of the tracking unit 5c generates N states of particles Pp by adding random noise to the position coordinates of the designated initial position for each specific part P of the subject S to be tracked. (Step S4).

Next, after the tracking unit 5c designates the start frame (first frame image F1) among the plurality of frame images F,... Constituting the moving image data D as a processing target frame (step S5), For each specific part P (for example, the left arm) to be tracked, one of the N particles Pp generated by the particle generation unit c1 is designated (step S6).
Subsequently, the difference calculation unit c2 of the tracking unit 5c calculates the difference DIFF between the specified particle Pp and the template Pt generated by the template generation unit 5b for each specific part P (for example, the left arm). (Step S7). For example, when the left arm is the specific part P, the difference calculation unit c2 calculates the difference UD and LD between the particle Pp for each of the upper arm and the forearm constituting the left arm, and calculates the calculated difference between the upper arm and the forearm. UD and LD are added to calculate the difference DIFF from the particle Pp.
The difference calculation unit c2 temporarily stores the calculated difference DIFF for each specific part P in association with the identification information of the particle Pp in a predetermined storage unit (for example, the memory 2) (step S8). ).

Next, the tracking unit 5c determines whether or not the processing for calculating the dissimilarity DIFF has been performed for all the particles Pp generated by the particle generation unit c1 (step S9).
Here, if it is determined that all the particles Pp are not processed (step S9; NO), the tracking unit 5c is the next unprocessed next among the N particles Pp generated by the particle generation unit c1. Particle Pp is newly designated (step S10). Thereafter, the tracking unit 5c returns the process to step S7, and executes the processes of steps S7 and S8 in the same manner as described above for the newly designated particle Pp.
As a result, the degree of difference DIFF of the newly designated particle Pp is calculated for each specific part P and temporarily stored in a predetermined storage means (for example, the memory 2).
Each of the above processes is repeatedly executed until it is determined in step S9 that all the particles Pp have been processed.

On the other hand, if it is determined in step S9 that all the particles Pp have been processed (step S9; YES), the tracking processing unit 5 immediately before the frame image F to be processed, as shown in FIG. It is determined whether or not the processing for calculating the degree of difference has been performed for the frame image F (step S11).
Here, when the processing target is the start frame of the moving image data D, it is determined that the previous frame image F has not been processed (step S11; NO), and the tracking processing unit 5 performs the particle likelihood correction processing. (Step S12) is skipped, and it is determined whether or not the process of tracking the specific part P is performed for all the frame images F1 to Fn constituting the moving image data D (Step S13).
If it is determined that all the frame images F1 to Fn have not been processed (step S13; NO), the tracking unit 5c is not processed among the plurality of frame images F,. The next frame image F (for example, the second frame image F2) is newly designated (step S14).

Next, the particle generation unit c1 adds random noise to each of the N particles Pp for each specific part P (step S15), and then returns the process to step S6 to newly specify a frame image. For F (for example, the second frame image F2), the processes in steps S6 to S10 are executed in the same manner as described above.
As a result, the difference DIFF of the particles Pp is calculated for each specific part P of the newly designated frame image F (for example, the second frame image F2).

  Thereafter, if it is determined in step S11 that the previous frame image F (for example, the first frame image F1 with respect to the second frame image F) has been processed (step S11; YES), tracking is performed. The processing unit 5 performs a particle likelihood correction process (see FIG. 4) (step S12).

<Particle likelihood correction processing>
Here, the particle likelihood correction processing will be described with reference to FIG.
FIG. 4 is a flowchart illustrating an example of an operation related to the particle likelihood correction process.

  As shown in FIG. 4, first, the particle selection unit 5 d acquires the difference DIFF of N particles Pp from a predetermined storage unit (for example, the memory 2) for each specific part P, and these N Among the particles Pp, K particles (K is a natural number smaller than N) are selected in order from the smallest difference DIFF (step S21).

Next, the path search unit 5e adds the selected K particles Pp to the nodes of the weighted directed graph G for each specific part P (step S22), and the difference DIFF of each particle Pp is one. Based on the inter-coordinate distance (Euclidean distance D) with the previous frame image F, the route cost E of the edge connecting the nodes is set (step S23). For example, when the specific part P is the left arm, the route search unit 5e reads the frame image F to be processed and the previous frame image F (for example, the second and first frame images F1, F2, etc.)) The Euclidean distance D between the position coordinates of the left shoulder, left elbow, left wrist, etc. in the left arm is multiplied by the adjustment parameter α, the difference DIFF of the particles Pp related to the second frame image F is added, and the edge Route cost E is calculated.
Subsequently, for each specific part P, the route search unit 5e performs processing on the frame image F (for example, the second frame) from the first frame image F1 for the K (for example, four) particles Pp. The arrival cost of the route to the image F2 etc. is calculated (step S24).

Thereafter, the likelihood correcting unit 5f corrects the likelihood of each particle Pp based on the arrival cost of the path for each particle Pp calculated by the path searching unit 5e, and sets N particles Pp (step) S25). For example, the likelihood correcting unit 5f sets the ratio of the reciprocal of the calculated arrival cost of the path for each particle Pp as the likelihood of the particle Pp, and multiplies the reciprocal of the path cost of each particle Pp by N. And a total of N particles Pp are set.
Thereby, the particle likelihood correction process is terminated.

Returning to FIG. 3, if it is determined in step S13 that all the frame images F1 to Fn have not been processed (step S13; NO), the tracking unit 5c moves the process to step S14, and the moving image data Among the plurality of D frame images F,..., A new unprocessed next frame image F (for example, the third frame image F3) is newly designated (step S14).
Next, the particle generation unit c1 adds random noise to each of the N particles Pp whose likelihood has been corrected for each specific part P (step S15), and then returns the process to step S6 to newly In the same manner as described above, the processes in steps S6 to S12 are performed on the frame image F specified in (3) (for example, the third frame image F3).
As a result, the likelihood of the particle Pp is corrected for the third frame image F3.
Each of the above processes is repeatedly executed until it is determined in step S13 that all the frame images F1 to Fn have been processed.

On the other hand, if it is determined in step S13 that all the frame images F1 to Fn have been processed (step S13; YES), the tracking result specifying unit 5g has K (for example, four, etc.) in the final frame image Fn. The shortest path (see FIG. 6C) that minimizes the arrival cost for the particle Pp is identified (step S16). Specifically, the tracking result specifying unit 5g calculates the arrival cost among the plurality of routes based on the calculation result of the arrival cost of the plurality of routes for each particle Pp for the final frame image Fn by the route search unit 5e. Specify the shortest path that is the smallest. Then, the tracking result specifying unit 5g specifies each particle Pp of each frame image F related to the shortest path as a tracking result of the specific part P (step S17).
The tracking result specifying unit 5g stores the calculated tracking result of each specific part P in a predetermined storage unit (for example, the memory 2) (step S18), and ends the subject tracking process.

As described above, according to the subject tracking device 100 of the present embodiment, the specific part of the subject S for one frame image F to be processed among the plurality of frame images F,. A predetermined graph (for example, a weighted directed graph) by dynamic programming based on the degree of difference between the plurality of particles Pp generated for each P,... And the corresponding region in the frame image F before the one frame image F. G) A route search is performed, and the likelihood of the plurality of particles Pp,... Of the specific part P in one frame image F is corrected based on the search result. ..., the specific part P can be tracked with high accuracy in the frame image F after the one frame image F. That is, the tracking result using the particle filter of the specific part P in the one frame image F to be processed is reflected in the tracking using the particle filter of the specific part P in the frame image F after the one frame image F. As a result, it becomes unnecessary to widen the distribution range of the particles Pp that cause extra noise or increase the number of particles Pp in the subsequent processing of the frame image F, and to improve the robustness of the processing using the particle filter. It is possible to achieve stable tracking that is less susceptible to disturbances and the like.
Then, after the likelihood of the plurality of particles Pp,... Of the specific part P related to each of the frame images F before the final frame image Fn is corrected, a plurality of values are obtained based on the tracking result of the specific part Pp of the final frame image Fn. The tracking result of each specific part P of the frame images F,. Specifically, the tracking result of each specific part P of the plurality of frame images F,... Can be specified based on the arrival cost of the path for each particle Pp from the first frame image F1 to the final frame image Fn. it can. Accordingly, it is possible to search for a path through the particle Pp in an optimal state in each frame image F in consideration of the time axis of the plurality of frame images F,..., And as a result, each frame related to the identified path. The particle Pp of the image F can be specified as the tracking result of the specific part P.
Thus, the tracking process of the specific part P which is highly accurate and excellent in stability can be realized.

  In addition, for one frame image F to be processed, a predetermined number (for example, K) of a plurality of (for example, N) particles Pp for each specific part P with reference to the degree of difference from the template Pt of the specific part P. ) And a route search is performed based on the degree of difference between the selected particle Pp and the corresponding region in the frame image F before the one frame image F. Therefore, the processing target frame image F The predetermined number of particles Pp used for the route search by the dynamic programming method can be appropriately selected in consideration of the state of the plurality of particles Pp,... It can be done properly.

  Further, for each of a plurality of particles Pp,... Of a specific part P of one frame image F to be processed, a predetermined reference frame image F to a single frame image F of the plurality of frame images F,. The arrival cost of the route is calculated by dynamic programming, and the likelihood of each particle Pp is corrected based on the calculated arrival cost of the route for each particle Pp. Therefore, the plurality of specific parts P in one frame image F are corrected. The likelihood of the particles Pp,... Can be appropriately corrected in consideration of the arrival cost.

  Further, with respect to one frame image F to be processed, a predetermined graph is obtained by dynamic programming based on the degree of difference between the plurality of particles Pp,... Generated for each specific part P and the template Pt of the specific part P. Since the route search is performed above, the degree of difference between the plurality of particles Pp,... And the corresponding region in the frame image F before the one frame image F, that is, between the two adjacent frame images F, F. In addition to the state change amount, it is possible to appropriately perform a route search by dynamic programming in consideration of the state of a plurality of particles Pp,... In one frame image F to be processed.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, when the specific part P is sequentially tracked from each of the frame images F constituting the moving image data D imaged by the imaging device, the tracking state of the specific part P may be notified. That is, for example, the state of the minimum path cost obtained by the dynamic programming method by processing the latest frame image F being captured is displayed in real time on the display screen, and after all the capturing is completed, the last frame is displayed. You may make it display the tracking process by the path | route obtained by the process of the image F. FIG.

  Furthermore, the configuration of the subject tracking apparatus 100 is merely an example of the configuration illustrated in the above embodiment, and is not limited thereto. For example, the subject tracking device 100 is exemplified by a computer such as a workstation. However, the subject tracking device 100 is not limited to this and may be configured by a general personal computer or an imaging device.

In addition, in the above-described embodiment, the functions of the acquisition unit, the tracking unit, the search unit, the correction unit, and the specifying unit are controlled under the control of the central control unit 1, and the moving image acquisition unit 5a, the tracking unit 5c, The configuration is realized by driving the route searching unit 5e, the likelihood correcting unit 5f, and the tracking result specifying unit 5g. However, the configuration is not limited to this, and a predetermined program or the like is executed by the CPU of the central control unit 1 It is good also as a structure implement | achieved by doing.
That is, a program memory (not shown) that stores a program stores a program including an acquisition processing routine, a tracking processing routine, a search processing routine, a correction processing routine, and a specific processing routine. Then, the CPU of the central control unit 1 may function as a means for acquiring a moving image of the subject S by an acquisition processing routine. Further, the CPU of the central control unit 1 tracks the specific portion P of the subject S by processing using a particle filter in each of the plurality of frame images F,... Constituting the acquired moving image by the tracking processing routine. You may make it function as a means. Further, the CPU of the central control unit 1 in the search processing routine selects any one of the plurality of particles Pp,... Generated for each specific part P for any one of the plurality of frame images F,. You may make it function as a means to search a path | route on a predetermined | prescribed graph by a dynamic programming method on the basis of the difference with the corresponding area | region in the frame image F before the one frame image F. FIG. Further, the CPU of the central control unit 1 is caused to function as a means for correcting the likelihood of the plurality of particles Pp of the specific part P in any one of the frame images F based on the search result by the correction processing routine. May be. Further, the CPU of the central control unit 1 performs the likelihood of the plurality of particles Pp,... Of the specific part P related to each of the frame images F before the final frame image Fn of the plurality of frame images F by the specific processing routine. After the correction of, based on the tracking result of the specific part P of the final frame image Fn, it may function as means for specifying the tracking result of each specific part P of the plurality of frame images F,.

  Similarly, the selection unit may be configured to be realized by executing a predetermined program or the like by the CPU of the central control unit 1.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Acquisition means for acquiring a moving image of a subject;
In each of the plurality of frame images constituting the moving image acquired by the acquisition unit, a tracking unit that tracks a specific part of the subject by processing using a particle filter;
Regarding any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a corresponding region in a frame image before the one frame image, Search means for performing a route search on a predetermined graph by dynamic programming based on the difference between
Based on the search result by the search means, a correction means for correcting the likelihood of the plurality of particles in the specific part in the one frame image,
After correcting the likelihood of the plurality of particles in the specific part of each of the frame images before the final frame image of the plurality of frame images by the correcting unit, the tracking unit calculates the likelihood of the final frame image. A specifying means for specifying a tracking result of the specific part of each of the plurality of frame images based on a tracking result of the specific part;
A subject tracking device comprising:
<Claim 2>
The tracking means further includes:
The specific portion of the frame image after the one frame image is tracked based on the plurality of particles related to the one frame image whose likelihood is corrected by the correcting means. The subject tracking apparatus according to claim 1.
<Claim 3>
The search means further includes:
A path from a predetermined reference frame image of the plurality of frame images to the one of the frame images for each of the plurality of particles at the specific part of the frame image generated by the tracking unit. The arrival cost of
The correction means further includes:
The subject tracking device according to claim 1, wherein the likelihood of each particle is corrected based on the arrival cost of the path for each particle calculated by the search unit.
<Claim 4>
The specifying means further includes:
Based on the arrival cost of the path for each particle from the first frame image to the last frame image of the plurality of frame images calculated by the search means, the specific part of each of the plurality of frame images The subject tracking device according to any one of claims 1 to 3, wherein a tracking result is specified.
<Claim 5>
The search means further includes:
For each frame image, a route search is performed on a predetermined graph by dynamic programming based on the difference between the plurality of particles generated by the tracking unit for each specific part and the template of the specific part. The subject tracking device according to any one of claims 1 to 4, wherein the subject tracking device is characterized.
<Claim 6>
The search means further includes:
For each frame image, a plurality of particles generated by the tracking unit for each specific part are sequentially added to a weighted directed graph, and a route search is performed on the weighted directed graph by dynamic programming. The subject tracking device according to any one of claims 1 to 5.
<Claim 7>
Selecting means for selecting a predetermined number of particles from the plurality of particles generated by the tracking means for each specific part with respect to any one frame image on the basis of the degree of difference from the template of the specific part; In addition,
The search means includes
7. The route search is performed based on a degree of difference between a particle selected by the selection unit and a corresponding area in a frame image prior to any one of the frame images. The subject tracking device according to claim 1.
<Claim 8>
A subject tracking method using a subject tracking device,
An acquisition process for acquiring a moving image of a subject;
In each of the plurality of frame images constituting the moving image acquired by the acquisition process, a tracking process for tracking a specific part of the subject by a process using a particle filter;
For any one of the plurality of frame images, a plurality of particles generated by the tracking process for each specific part and a corresponding region in a frame image before the one frame image, Search processing for performing route search on a predetermined graph by dynamic programming based on the difference between
Based on the search result by the search process, a correction process for correcting the likelihood of the plurality of particles in the specific part in the one frame image,
After the correction of the likelihood of the plurality of particles of the specific part related to each of the frame images before the final frame image of the plurality of frame images by the correction processing, the final frame image of the final frame image by the tracking processing A specifying process for specifying a tracking result of the specific part of each of the plurality of frame images based on a tracking result of the specific part;
A method for tracking an object, comprising:
<Claim 9>
Subject tracking device computer
An acquisition means for acquiring a moving image of a subject;
A tracking means for tracking a specific part of a subject by processing using a particle filter in each of a plurality of frame images constituting the moving image acquired by the acquisition means;
Regarding any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a corresponding region in a frame image before the one frame image, Search means for performing a route search on a predetermined graph by dynamic programming based on the difference between
Correction means for correcting the likelihood of the plurality of particles in the specific part in the one frame image based on the search result by the search means,
After correcting the likelihood of the plurality of particles in the specific part of each of the frame images before the final frame image of the plurality of frame images by the correcting unit, the tracking unit calculates the likelihood of the final frame image. Identification means for identifying the tracking result of the specific part of each of the plurality of frame images based on the tracking result of the specific part;
A program characterized by functioning as

100 Subject tracking device 1 Central control unit 5 Tracking processing unit 5a Moving image acquisition unit 5c Tracking unit 5d Particle selection unit 5e Path search unit 5f Likelihood correction unit 5g Tracking result identification unit

Claims (9)

Acquisition means for acquiring a moving image of a subject;
In each of the plurality of frame images constituting the moving image acquired by the acquisition unit, a tracking unit that tracks a specific part of the subject by processing using a particle filter;
Regarding any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a corresponding region in a frame image before the one frame image, Search means for performing a route search on a predetermined graph by dynamic programming based on the difference between
Based on the search result by the search means, a correction means for correcting the likelihood of the plurality of particles in the specific part in the one frame image,
After correcting the likelihood of the plurality of particles in the specific part of each of the frame images before the final frame image of the plurality of frame images by the correcting unit, the tracking unit calculates the likelihood of the final frame image. A specifying means for specifying a tracking result of the specific part of each of the plurality of frame images based on a tracking result of the specific part;
A subject tracking device comprising: The tracking means further includes:
The specific portion of the frame image after the one frame image is tracked based on the plurality of particles related to the one frame image whose likelihood is corrected by the correcting means. The subject tracking apparatus according to claim 1. The search means further includes:
A path from a predetermined reference frame image of the plurality of frame images to the one of the frame images for each of the plurality of particles at the specific part of the frame image generated by the tracking unit. The arrival cost of
The correction means further includes:
The subject tracking device according to claim 1, wherein the likelihood of each particle is corrected based on the arrival cost of the path for each particle calculated by the search unit. The specifying means further includes:
Based on the arrival cost of the path for each particle from the first frame image to the last frame image of the plurality of frame images calculated by the search means, the specific part of each of the plurality of frame images The subject tracking device according to any one of claims 1 to 3, wherein a tracking result is specified. The search means further includes:
For each frame image, a route search is performed on a predetermined graph by dynamic programming based on the difference between the plurality of particles generated by the tracking unit for each specific part and the template of the specific part. The subject tracking device according to any one of claims 1 to 4, wherein the subject tracking device is characterized. The search means further includes:
For each frame image, a plurality of particles generated by the tracking unit for each specific part are sequentially added to a weighted directed graph, and a route search is performed on the weighted directed graph by dynamic programming. The subject tracking device according to any one of claims 1 to 5. Selecting means for selecting a predetermined number of particles from the plurality of particles generated by the tracking means for each specific part with respect to any one frame image on the basis of the degree of difference from the template of the specific part; In addition,
The search means includes
7. The route search is performed based on a degree of difference between a particle selected by the selection unit and a corresponding area in a frame image prior to any one of the frame images. The subject tracking device according to claim 1. A subject tracking method using a subject tracking device,
An acquisition process for acquiring a moving image of a subject;
In each of the plurality of frame images constituting the moving image acquired by the acquisition process, a tracking process for tracking a specific part of the subject by a process using a particle filter;
For any one of the plurality of frame images, a plurality of particles generated by the tracking process for each specific part and a corresponding region in a frame image before the one frame image, Search processing for performing route search on a predetermined graph by dynamic programming based on the difference between
Based on the search result by the search process, a correction process for correcting the likelihood of the plurality of particles in the specific part in the one frame image,
After the correction of the likelihood of the plurality of particles of the specific part related to each of the frame images before the final frame image of the plurality of frame images by the correction processing, the final frame image of the final frame image by the tracking processing A specifying process for specifying a tracking result of the specific part of each of the plurality of frame images based on a tracking result of the specific part;
A method for tracking an object, comprising: Subject tracking device computer
An acquisition means for acquiring a moving image of a subject;
A tracking means for tracking a specific part of a subject by processing using a particle filter in each of a plurality of frame images constituting the moving image acquired by the acquisition means;
Regarding any one of the plurality of frame images, a plurality of particles generated by the tracking unit for each specific part and a corresponding region in a frame image before the one frame image, Search means for performing a route search on a predetermined graph by dynamic programming based on the difference between
Correction means for correcting the likelihood of the plurality of particles in the specific part in the one frame image based on the search result by the search means,
After correcting the likelihood of the plurality of particles in the specific part of each of the frame images before the final frame image of the plurality of frame images by the correcting unit, the tracking unit calculates the likelihood of the final frame image. Identification means for identifying the tracking result of the specific part of each of the plurality of frame images based on the tracking result of the specific part;
A program characterized by functioning as

Images