JP6065427B2 - Object tracking method and object tracking apparatus - Google Patents

Description

  The present invention relates to an object tracking method and an object tracking apparatus.

  Object tracking is one of the most important technologies in the field of image analysis and machine vision. In the object tracking, it is mainly difficult to associate objects in continuous video frames. There is a problem that it is difficult to track an object when it is faster than the rate, and when a certain degree of distortion occurs due to a change in the direction of the tracked object during the exercise period, the complexity of the problem is reduced. There is a difficulty that it increases more.

  Non-Patent Document 1, “Mathias Kolsch, Doctoral Dissertation,“ Vision Based Hand Gesture Interfaces for Wearable Computing and Virtual Environments ”, UNIVERSITY OF CALIFORNIA, Santa Barbara, 2004”, Flock Features in 2D (2D) Images (Flocks of features) has been proposed. In this method, a plurality of feature points are tracked by the KLT tracker based on the optical flow vector calculation, and the optical flow vectors are grouped according to the overall restriction loosely. Predicting position. In this method, the skin color plays an important role in feature grouping, and the skin color probability density function is used for feature point supplementation. For this reason, there is a problem in that the tracking result deviates from the target object when the hand moves and gives up the skin color approximate region (for example, face).

  Actually, in the conventional 2D tracking technique, the problem that the stability of the feature is inferior and the problem that the mismatch between the features tends to occur cannot be solved well. Compared to 2D object tracking technology, a 3D (3D) camera can give more depth information to each object in the 3D world, and the depth information has an approximate color and shape on the Z axis of the coordinate system Different objects can be distinguished.

  1A to 1D are diagrams showing image processing steps of a 3D camera. Here, a Prime Sense 3D camera will be described as an example.

  FIG. 1A shows a scene, and the Prime Sense 3D camera shown in FIG. 1B collects the scene of FIG. 1A to obtain the depth image shown in FIG. 1C. Each pixel point of the depth image includes actual target depth coordinate data (depth value) corresponding to the pixel point. For example, in the pixel matrix shown in FIG. 1D, each element in the matrix has a depth. Corresponding to one pixel point in the image, the value of the pixel represents the distance from the target surface corresponding to the pixel point in world coordinates to the camera, and the distance unit may be, for example, mm.

  The principle of the Prime Sense 3D camera's depth measurement technique is as follows. First, the camera shown in FIG. 1B uses an infrared light source, and a pattern point consisting of a group of invisible infrared rays is the object of the scene shown in FIG. After projecting on the surface and obtaining a pattern image after projection by the CMOS sensor, the processor calculates the depth value of each point on the target surface by the triangulation measurement technique from the deviation of the points in the pattern light. The depth image has no color information, but can be visualized by various methods, for example, the black and white gradation shown in FIG. 1C. By mixing the depth data stream and the image stream, a color 3D image can be generated, and an RGB image stream and a depth value stream corresponding to each can be output synchronously by the 3D camera.

  US 20100194741 A1, which is Patent Document 1, proposes an object tracking method using an optical flow in a depth image. In this method, each pixel point in the isolated region is assigned a black and white gradation value according to the depth value, thereby generating a “zebra” pattern in a monochrome gradation method. Thereafter, a new position of each pixel point in each region is determined by optical flow calculation. In this method, tracking is stabilized by speed prediction, but such a bottom-up processing method that relies only on the tracking result of a single point lacks a policy such as prevention of error propagation. According to the method, a stable tracking result cannot be obtained particularly in the case of long-time complicated motion tracking.

  In the prior art, a method for determining the scale of the object in the imaging plane during the object tracking period is not considered. Even if the prior art that considers the determination of the image scale exists, the target scale prediction means used is usually stochastic block matching. For example, in particle filter tracking, a random change to the current state (object And the weight of the block after conversion is calculated based on the relation coefficient between the block after the change and the block before the change, and the scale is determined from the average state after the calculation. Processing methods are unreliable and affect the accuracy of the final tracking results.

  In particular, in the case of man-machine interaction systems, the background is usually arbitrary and complex, and the movement of the object is also relatively complex, so that not only the movement direction and speed change, but also the movement object itself. There are also shape adjustments. In particular, how to obtain a reliable and stable tracking result in such an environment in continuous tracking over a long period of time is extremely important and challenging.

  The present invention has been made to solve the above-mentioned problems in the prior art. According to the present invention, an object can be tracked from a depth image sequence, the position of the object can be acquired, and further, a reduced scale in the imaging plane of the object can be obtained.

Means for solving the problem

  In order to solve the above problem, in the embodiment of the present invention, a 3DCCA (Three-Dimension Connected Component Analysis) technique is applied to a depth image to obtain a list of all communication areas, Finally, the object communication area related to the tracking object is determined. In 3DCCA, images can be grouped into a plurality of different communication areas by performing the same marking on adjacent pixel points according to the connectivity of the pixels, and any two pixels within the same communication area. There will be at least one D-communication path between the points. With 3DCCA on the depth image, targets having different depth values can be identified from a complex background environment, and a mask image of the object can be obtained.

  When the 3DCCA technique is used, the object can be efficiently separated from the complicated background, and the approximate position of the object can be predicted based on the movement history information of the object. Based on the prediction result of the object location communication area, the optical flow vector is obtained by tracking the feature point using the optical flow method in the object location communication area, and from the optical flow vector of a plurality of feature points, The final object position can be extracted.

  Furthermore, the 3DCCA and the depth information make it possible to determine the scale on the image plane of the object, that is, to determine the image scale. Based on the scale of the object in the initial state, the depth information in the initial state, and the current depth information, the image scale of the current object can be predicted from the approximate triangle theory.

  According to one aspect of the present invention, a communication area acquisition step of performing a three-dimensional communication area analysis on the input initial depth image and acquiring a communication area list of the initial depth image, and a known current state of the object in the initial depth image. An initial object determining step for determining a communication area where the object is located from the position and determining n feature points (n is a natural number) in an image portion corresponding to the communication area, and a subsequent input input after the initial depth image A tracking step for identifying the object communication area where the object is located from each candidate communication area obtained by performing a three-dimensional communication area analysis of the depth image and obtaining a communication area list of the subsequent depth image, and identified in the tracking step. An object positioning step of tracking the n feature points from the object communication area and updating the current position of the object; Having, object tracking method is provided.

  In another aspect of the present invention, a communication area acquisition device that performs a three-dimensional communication area analysis on an input initial depth image and acquires a communication area list of the initial depth image, and a known object of the initial depth image An initial object determination device that determines a communication area where an object is located from the current position and determines n feature points (n is a natural number) in an image portion corresponding to the communication area, and a subsequent input that is input after the initial depth image A tracking device that performs a three-dimensional communication area analysis of the depth image of the image and identifies the object communication area where the object is located from each candidate communication area obtained from the communication area list of the subsequent depth image; and An object positioning device that tracks the n feature points from the identified object communication area and updates the current position of the object. Object tracking apparatus is provided.

  In the embodiment of the present invention, the communication area where the tracking target is located is divided by 3DCCA (three-dimensional communication area analysis) technology, and the optical flow tracking result of each feature point is used as a reference standard for evaluation. The embodiments of the present invention can be applied to various man-machine interaction application systems, and can be applied to technical directions such as man-machine interaction games and virtual reality remote control.

  According to the embodiment of the present invention, the object tracking problem in the man-machine interaction application by the depth camera can be solved, and the execution of the embodiment of the present invention has no special requirement for the object to be tracked, and has a special purpose. Marks and gloves are not required, and it can be applied not only to objects with clear outlines but also to non-fixed objects, enabling real-time processing and obtaining stable and reliable tracking results.

FIG. 1A is a diagram illustrating a scene in an image processing process of a 3D camera. FIG. 1B shows a 3D camera that collects the scene of FIG. 1A. FIG. 1C shows a depth image of the scene of FIG. 1A. FIG. 1D is a pixel matrix of the depth image in FIG. 1C. It is a whole flowchart of the object tracking process in the Example of this invention. It is a figure which shows a depth image. It is a 3DCCA execution result to the depth image shown by FIG. 3A. It is a figure which shows the object determination result from an image. It is a basic principle diagram of scale reduction calculation on an imaging plane of an object in a tracking stage. It is a head object image scale change figure at the time of head object tracking. It is another head object image scale change figure at the time of head object tracking. It is a flowchart of the object exact positioning step in the Example of this invention. It is the figure which showed the tracking result in case a tracking target is a hand. It is the figure which showed the other tracking result in case a tracking target is a hand. It is a block diagram of the object tracking apparatus in the Example of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 2 is an overall flowchart of the object tracking method in the embodiment of the present invention. As shown in FIG. 2, the object tracking method performs a three-dimensional communication area analysis on the input initial depth image and acquires a communication area acquisition step S100 for acquiring a communication area list of the initial depth image, An initial object determining step S200 for determining a communication area where the object is located from a known current position of the object in the depth image, and determining n feature points (n is a natural number) in the image portion corresponding to the communication area; Tracking that identifies the object communication area where the object is located from each candidate communication area obtained by performing a three-dimensional communication area analysis of the subsequent depth image input after the initial depth image and obtaining the communication area list of the subsequent depth image The n feature points are tracked from the object communication area identified in step S300 and the tracking step, and the object Having an object positioning step S400 to update the current position.

  The depth images processed in the embodiments of the present invention are input by various known input techniques, for example, read from various depth image collection devices or storage devices, or obtained by a network. It is output by a known output technique, for example, directly converted into control information, stored in various storage devices, output via a network, or output by a printer.

  The communication area acquisition step S100 and the initial object determination step S200 described above can be regarded as an initialization stage. In the initialization stage, the first one depth image in the depth image sequence is selected and processed. Even when an arbitrary depth image is selected and the initialization stage operation is performed, the depth image to be operated in the initialization stage can be referred to as an initial depth image.

  The tracking step S300 and the object positioning step S400 described above can be roughly regarded as a tracking stage, and each depth image to be a subsequent depth image after the initial depth image is repeatedly processed sequentially or at regular intervals. All depth images processed each time can be regarded as the current depth image, and the previous processing result used is called the previous result, and the previous processing may target the subsequent depth image. The initial depth image may be the target (when processing the first subsequent depth image).

  In the communication area acquisition step S100, a list of CCs (communication areas) included in the depth image is acquired by the 3DCCA means. Regardless of the initialization stage or the tracking stage, a 3DCCA operation on the depth image is first required to obtain a list including all communication area information in the figure. In 3DCCA (that is, three-dimensional communication area analysis), proximity points are detected in the X-axis and Y-axis directions of the image coordinate system with respect to the input depth image, and the proximity on the Z-axis is within a certain range. Since the same numerical numbering is performed on the point pixels and the pixels having the same number form one communication area, the output of 3DCCA is a set of connected shape components. By the 3DCCA operation of the depth image, the pixel level depth information obtained from the 3D camera is integrated into a small number of target sets, and by using these target sets, determination of targets at different depths on the Z-axis, Analysis of scene content becomes possible.

  Embodiments of the present invention can use the following specific 3DCCA implementation, which is based on the commonly used 2DCCA, with modifications to apply 3D image data. It is obtained.

  First, 3DCC (three-dimensional communication area) is defined as follows.

If the projections on the XY plane of two 3D points are close and the change in depth is less than a certain threshold D_TH, the two points are D-connected,
For two 3D points P and Q, there is a group of 3D point lists (P, p1, p2,... PN, Q), and any two adjacent points in the list are D-connected. And there is a D-communication path between the two points,
In a D-communication set of one 3D point, for each point p in the set, if there are no p neighboring points added to the communication set in a state satisfying the communication condition of the set on the XY plane, the D- The communication set is a maximum D-communication set, that is, a D-communication area.

  The method for searching for the D-communication area, that is, the 3DCCA method for searching for the maximum CC is as follows.

  1. Each point (x, y) is assigned the number of the communication area (CC) to which it belongs, and is denoted as LABEL (x, y).

  2. Define the threshold D_TH for the depth difference.

  3.1 Define a matrix data structure (first-in first-out) and write QUEUE.

  4). Initialize all points LABEL (x, y) to -1.

  5. Set the current communication area number cur_label to 1.

  6). It starts searching for the CC start point p-start where the next LABEL is -1. If no such point exists, the iteration stops.

  7). Set LABEL (p_start) to cur_label.

  8). Put all points p_start into the matrix QUEUE.

  9. If QUEUE is not empty, repeat the following steps.

    a. Remove the head node p_head (x, y) from the matrix.

b. For m proximity points of p_head,
If i.LABEL (k)> 0 (k is the index value of m proximity points), go to the next proximity point,
ii. If the difference between the depth of the proximity point of k and the depth of p_head is D_TH, the kth proximity point is put into the matrix and LABEL (k) is set to cur_label.

  10. Increase cur_label by 1 and repeat from step 6.

  In the above method, the proximity point of the point (x, y) is defined as a point whose coordinates are as follows: (x-1, y-1), (x-1, y), (x-1 , y + 1), (x, y-1), (x, y + 1), (x + 1, y-1), (x + 1, y), (x + 1, y + 1). If the coordinate position of the proximity point exceeds the image range (a negative value or exceeds the image resolution), no processing is performed.

  3A and 3B show the results of 3DCCA execution on a depth image. FIG. 3A shows the depth image, and the grayscale values of the respective pixel points in the drawing represent the perspective information of the respective depths. FIG. 3B shows a result of 3DCCA execution on the depth image shown in FIG. 3A, and each obtained communication area is divided by each gradation (color). Thus, when the hand is extended, as shown by the rectangular frame Q1 in FIG. 3B, the hand can be surely divided from the body and the environment by 3DCCA.

  The execution efficiency of 3DCCA can be improved by optimizing the algorithm, and further, higher efficiency can be obtained by applying 3DCCA to the local region according to actual demand.

  Since the initial object determination step S200 is an initialization stage, the object position can be designated by external input, and the object can be easily surrounded by a square frame, or the object can be detected and identified by a real-time detection operator. . At this stage, since the position of the object is known in advance, the CC where the object is located can be easily determined from the CC list associated with the object.

  Note that, for example, the CC where the object is located may be automatically determined by enlightenment rules such as the nearest maximum communication area principle.

  FIG. 4 is a diagram illustrating an object determination result from an image. In FIG. 4, a white area of the rectangular frame Q2 represents a communication area associated with an object when the object is a hand. Since it is an initial stage, if the purpose is to divide the hand object in FIG. 4, the rectangular frame Q2 is drawn in FIG. 4, and the local 3DCCA operation may be performed within the area of the rectangular frame Q2. The rectangular frame Q2 in FIG. 4 has two CC regions of the hand CC and the background CC after 3DCCA, and the hand CC having the closest area is the CC where the object to be tracked is located.

  After determining the CC where the object is located, a mask image of the object can be obtained. A gradation image corresponding to a mask image (a gradation image at this time indicates an image obtained after visualization of a depth image, and the optical flow can be traced by the gradation image) and a mask image From the RGB color image synchronized with the depth image, n feature points for subsequent optical tracking are selected. That is, the n feature points are extracted from the depth image where the corresponding image portion is located or the color image synchronized with the depth image.

  The feature point at this time indicates a corner having a relatively large response value in a gradation image (an RGB color image can be converted into a gradation image by a color removal operation), for example, a harris corner. . The mutual distance between n feature points is equal to or greater than a certain threshold value (first threshold value), and the minimum distance between the two feature points is set so that the space distance between any two feature points is not too close. By limiting, it is possible to ensure the effectiveness of each tracking result. In other words, a mutual interval between the n feature points is equal to or greater than a first threshold value, and each of the n feature points is a corner that is distinguished from an adjacent pixel point by a specific tracking operator.

  Feature points that meet the requirements can be selected using the function GoodFeaturesToTrack in the open source item OpenCV.

  The object process then proceeds to the tracking stage.

In the tracking step, first, a 3DCCA operation is performed on each depth image (subsequent depth image) input after the initial depth image to obtain a CC list of each depth image, and then an object CC is determined. Specifically, in the tracking step S300, the degree of similarity is highest with the result obtained by adding the state change after the motion prediction to the previously determined position of the object from each candidate communication area in the communication area list of the subsequent depth image. The candidate communication area is searched for as the object communication area where the object is currently located.
A 3DCCA operation is performed on the currently processed depth image in the array of subsequent depth images by means similar to the means used for the initial depth image to obtain a CC list of the current depth image. However, in the determination process of the object CC, unlike the initialization stage, the position of the object CC is unknown at this time, and all communication areas in the CC list become object candidate communication areas. The determination of the CC of the object location in the tracking step S300 is performed by searching for one CC from the CC list, which is most similar to the result of adding the state change after motion prediction to the previous location of the tracked object. It should have features, that is, it should have the highest similarity.

  The similarity is a position difference (motion velocity vector difference) between a candidate communication area position and a predicted position obtained after adding a predicted motion displacement in the x-axis, y-axis, and z-axis directions to the previous determined position of the object. Is equal) and the area difference between the candidate communication area and the previous determined object communication area.

Here, the search of the object CC can be regarded as the identification of the object CC and the rough positioning prediction of the object CC. With 3DCCA on the depth image, pixel level depth information is aggregated into the list of objects CC, making the process easier than single identification at the pixel level of the object. The CC where the object is located can be searched by simple means. For example, the CC closest to the previous position to the object CC can be searched by the following distance calculation formula (1).

  Where ||. || is the Euclidean norm operator, n is the index value of each frame depth image in the depth image sequence, and i is currently processing the nth frame depth image, i is the current Is the index value of the candidate communication area obtained from the nth frame depth image of V, and V (n-1) is the x-axis, y-axis, z of the object tracking result of the previous (n-1) th frame depth image. Vi (n) is the velocity vector in the x-axis, y-axis, and z-axis directions when the i-th candidate communication area is the object communication area in the case of the current n-th frame depth image. , A (n-1) is the area of the object communication area in the previous (n-1) th frame depth image, and Ai (n) is the area of the i th candidate communication area of the current nth frame. Yes, a is a weight, determined from user experience and test statistical analysis results, Di is the current i-th candidate CC and the previous o It is a metric distance of similarity between objects CC.

  It can be determined that the smaller the value of Di, the higher the similarity between the candidate communication area and the previous object communication area, and the i-th candidate communication area having the smallest Di value is the current object communication area. It becomes an area.

  In the object communication area search process, the object states considered include not only the motion velocity vectors in the x-axis, y-axis, and z-axis directions and the CC area associated with the object, but also the floors associated with other CCs. A feature amount such as a tone or a color histogram of a color image may be included.

  Note that the object CC may be determined by other means, and may be realized by a machine learning method, for example. Specifically, the object CC may be identified by training the classifier. In the classifier training process, the velocity vector, the communication area, and other feature quantities may be used as input during classifier training.

  After the tracking step S300, the object CC obtains a rough prediction of the object position from the target level list, and an image scale determining step for determining the image scale of the object in the initial depth image of the object identified in the tracking step. Furthermore, you may have. Calculation of the current tracked object image scale facilitates a more accurate description of the current state of the object. In particular, if the tracking object moves to a position at the same depth as the background, the object becomes a local part in the large range CC, and if it cannot be completely isolated from the surroundings, which is clearly distinguished from the surroundings, the object from the large CC It is necessary to calculate an object image scale for demarcating the location CC.

  In order to calculate the scale change on the imaging plane of the object in the tracking stage, it is necessary to calculate the average depth value of the object CC, and to record the average depth in the initial state and the image scale of the object in the initial state. Necessary.

FIG. 5 is a basic principle diagram for calculating the scale on the object imaging plane in the tracking stage.
As shown in FIG. 5, the object Obj is placed on the left side of the camera Camera, the average depth in the depth image of the first frame is d0, and the nth frame in a subsequent frame (n is a natural number other than 1) ) The average depth in the depth image is dn. The right side of the camera Camera is the imaging plane Plan, and when the distance of the object Obj to the camera Camera is d0 (average depth d0), the image scale on the imaging plane is S0, and the distance of the object Obj to the camera Camera is dn When (average depth dn), the image scale on the imaging plane is Sn. The actual scale of the object Obj is H, and when the distance of the object Obj to the camera Camera is d0, the projection scale at the dn of the object Obj is L (projection using the camera position as a point light source). Thus, according to the approximate triangle principle, S0 / Sn = L / H and L / H = dn / d0, so that S0 / Sn = dn / d0 reduces the image on the image plane at the object location. Equation (2) for calculating the scale can be derived.

  In the image scale determination step, an image scale is calculated by Sn = d0 / dn * S0 in Equation (2), where dn is the average of the object communication area where the object is located identified in the tracking step. Depth, d0 is the average depth of the object coverage of the determined object in the initial depth image, S0 is the image scale of the object in the initial depth image, and Sn is the subsequent depth image of the object Is an image scale.

  When the case shown in FIG. 5 is calculated by Equation (2), and dn in FIG. 5 is twice d0, the image scale Sn of the object on the dn position is half of the image scale S0 at the d0 position. It becomes. Needless to say, the calculation result of the formula (2) is consistent with human intuition.

  6A and 6B are head object image scale change diagrams when tracking a head object, FIGS. 6A and 6B represent different frames, a rectangular frame Q3 in FIG. 6A represents a tracking target, and FIG. The rectangular frame Q4 in FIG. 6 also represents the tracking target, but the tracking target shown in FIG. 6A has a change in image scale in FIG. 6B.

  By predicting the object communication area, not only the approximate position of the object and the image scale in the imaging plane can be predicted, but also the approximate shape description of the object can be made, and the tracking of the non-fixed object becomes easy.

  Next, in the object positioning step S400, a more accurate prediction of the object positioning point can be obtained by, for example, an optical flow method (in the image coordinates, the object scale is omitted and the object positioning point is abstracted into one positioning point. Represents the position of the object in the image). In the embodiment of the present invention, information on the object level is used for accurate prediction of the object position. Such information includes not only the shape of the object communication area but also the optical flow of a plurality of points in the object communication area. Contains object-level information that is commonly represented by tracking and intermediate value extraction.

  FIG. 7 is a flowchart of the object accurate positioning step in the embodiment of the present invention.

  As shown in FIG. 7, in the object positioning step S400, the feature point tracking step S420 for tracking the n feature points by the KLT tracker from the subsequent depth image, and the mask image information of the object communication area And class quantization of each tracked feature point, class quantization step S440 for weighting each tracked feature point, grouping the tracked feature points, and calculating the center point of the group, And a grouping step S460 for updating the current position of the object.

  In the feature point tracking step S420, each feature point is tracked by the KLT tracker, and a new position point on the subsequent depth image that is currently processed of each feature point is acquired. For example, the optical flow vector of the feature point is calculated by the OpenCV function cvCalcOpticalFlowPyrLK, and the corresponding new position of the feature point on the subsequent depth image is acquired.

  After the feature point tracking step S420, and before the class quantization step S440, a first removal step of removing feature points in the tracking feature points whose feature point tracking error exceeds a second predetermined threshold. It has further. For example, when calculating the optical flow vector of a feature point using the OpenCV function cvCalcOpticalFlowPyrLK, the maximum error parameter (second threshold) value can be set to 1150, and the error parameter exceeds this value. By removing the results, one can first remove tracking feature points with high or low error during feature point optical flow tracking. It goes without saying that 1150 is merely an example of the second threshold value, and the second threshold value may be other values such as 1100 and 1200, for example.

  In the class quantization step S440, each tracking feature point is subjected to class quantization based on the mask image information of the object communication area, and when the first removal step is performed, each remaining tracking feature after the first removal is performed. Point weight calculation is performed. As a simple processing method to be used, the weight is set to 1 if the new position of the tracking feature point is within the predicted area of the object communication area, and 0 otherwise.

  After the class quantization step S440 and before the grouping step S460, a second removal step of removing a predetermined proportional number of feature points having the largest interval between the tracking feature points and the center of gravity of the tracking feature points is further performed. Have.

を算出し、最後に、i=1……mにおいて、Dtiが最小値となる時の特徴点位置を探し、最終群分けの中心点とする。このときの最終群分けの中心点位置を前記オブジェクトの現位置とする。 In the grouping step S460, the total distance is the shortest to each feature point due to the existing tracking feature points (or the remaining tracking feature points after the second removal operation when the second removal step is performed). The center point of the group can be calculated, and the center point of the group can be used as the object positioning point for object tracking relatively accurately. As the grouping implementation method used, for example, the tracking feature points of the remaining m (m is a natural number) are numbered, such as P1, P2, P3, ..., Pm, and the i-th point (i is Is the index value, and 1 ... m), and the total length of the distance from the i-th feature point to each other feature point

Finally, the feature point position when Dti is the minimum value is searched for i = 1... M, and is set as the center point of the final grouping. The center point position of the final grouping at this time is set as the current position of the object.

  Since the number of tracking feature points may decrease after each operation in each tracking cycle, the number of feature points necessary for tracking the depth image of the next frame can be obtained by supplementing the feature point set with new feature points. Can be satisfied.

  In other words, after the grouping step S460, the total number of feature points of the object is n, and the interval between the n feature points that are located in the object and are replenished is equal to or greater than the first predetermined threshold. The replenishment step of replenishing new feature points may be further included.

  In the specific optical flow tracking method described above, the shape information of the predicted object communication area is sufficiently used, the optical flow information of the pixel points obtained from the lower layer is used, and the target level of the upper layer is also measured. Information can be combined to ensure the accuracy of tracking results, particularly the reliability of tracking results for non-fixed objects.

  8A and 8B are diagrams illustrating tracking results when the tracking target is a hand. In FIG. 8A, a rectangular frame Q5 represents the hand of the tracking target, a tracking feature point is represented by a plurality of minute points e1 as an example, and a target positioning point prediction result is represented by a relatively large point T1. In FIG. 8B, a rectangular frame Q6 represents the hand of the tracking target, a tracking feature point is represented by a plurality of minute points e2 as an example, and a target positioning point prediction result is represented by a relatively large point T2.

  The present invention can be further implemented as an object tracking device that executes the object tracking method described above. FIG. 9 is a block diagram of the object tracking device in the embodiment of the present invention.

  As shown in FIG. 9, the object tracking apparatus according to the embodiment of the present invention performs a three-dimensional communication area analysis on the input initial depth image, and acquires a communication area list of the initial depth image. From the communication area acquisition device 100 that executes the communication area acquisition step S100, and the known current position of the object in the initial depth image, the communication area where the object is located is determined, and n feature points in the image portion corresponding to the communication area (N is a natural number) for determining the initial object determination device 200 for executing the initial object determination step S200, and performing a three-dimensional communication area analysis of the subsequent depth image input after the initial depth image, The tracking step for identifying the object communication area where the object is located from each candidate communication area from which the communication area list of the depth image is acquired. A tracking device 300 that executes the step S300, and an object that executes the object positioning step S400 that tracks the n feature points from the object communication area identified by the tracking device 300 and updates the current position of the object. Positioning device 400.

  Here, an interval between the n feature points is equal to or greater than a first predetermined threshold value, and each of the n feature points is distinguished from a corner of an adjacent pixel point in a specific tracking operator. Also good.

  Further, the corresponding image portion from which the n feature points are extracted may be located in a depth image or a color image synchronized with the depth image.

  The tracking device 300 has the highest similarity with the result obtained by adding the state change after motion prediction to the previously determined position of the object from each candidate communication area in the communication area list of the subsequent depth image. To the object communication area where the object is currently located.

  The object tracking device according to an embodiment of the present invention further includes an image scale determination device that executes the image scale determination step of determining an image scale of an object in an initial depth image of the object communication area identified by the tracking device 300. Have.

  Here, the image scale determination device sets the average depth of the object communication area of the object location identified by the tracking device 300 to dn, and sets the average depth of the object communication region of the object location determined from the initial depth image to d0. And the image scale of the object in the initial depth image is S0, and the image scale of the subsequent depth image of the object is Sn, the image scale may be calculated by Sn = d0 / dn * S0. .

  Here, the object positioning apparatus 400 performs tracking of the n feature points by the KLT tracker from the subsequent depth image, the feature point tracking apparatus that performs the feature point tracking step S420, and an object communication area mask. Performs class quantization of each tracked feature point from image information, weights each tracked feature point, performs the class quantization step and classifies the tracking feature points A grouping device that calculates the center point of the group and updates the current position of the object, and performs the grouping step S460.

  Here, between the feature point tracking device and the class quantizing device, the feature points whose tracking point has an error during tracking of the feature point exceeds a second predetermined threshold are removed. A first removal device that executes one removal step, and a predetermined proportionality between the class quantization device and the grouping device that maximizes the distance between the tracking feature point and the center of the tracking feature point. You may make it also have the 2nd removal apparatus which performs the 2nd removal step which removes a number of feature points.

  In the object tracking device according to the embodiment of the present invention, after the grouping step, the total number of feature points of the object is n, and the mutual interval of the n feature points after being replenished is positioned in the object. A replenishing device that replenishes new feature points so as to be equal to or higher than the first predetermined threshold and that executes the replenishment step may be further included.

  In the object tracking method and the object tracking device according to the embodiment of the present invention, the three-dimensional communication area analysis to the depth image is performed, the entire communication area list is acquired, the motion of the communication area is predicted, and the object is detected from the communication area list. Each tracking feature point in the feature grouping process is identified by identifying the communication area where it is located, predicting the image scale on the image plane of the object, and providing reliable target level information (object communication area) It is possible to provide reference information important to the evaluation of the feature points, and to make it possible to implement feature point supplementation in optical flow tracking of a plurality of feature points. Propagation of errors can be efficiently prevented and a stable tracking result can be obtained by dividing the communication area and mutually supplementing the tracking of multiple points.

  In the embodiment of the present invention, as compared with Non-Patent Document 1, the object is divided by the 3DCCA operation to the depth map and the continuity of motion, so that the influence of color interference in 2D tracking can be eliminated, More reliable tracking results can be obtained.

  In the embodiment of the present invention, compared with Patent Document 1, the rough position is first roughly predicted by the determination of the communication area associated with the object, and the optical flow method is used more accurately under the rough prediction. Has gained a good position. The embodiment of the present invention combines the whole and local information, and uses up-to-bottom and bottom-to-up processing, so that stable tracking is better than the method in Patent Document 1. It can be easily realized. Note that shape edge information is obtained from the communication area, and tracking of multiple feature points whose mutual intervals are equal to or larger than a predetermined value is used in the image corresponding to the communication area mask to reflect the shape change of the object being tracked. Therefore, according to the embodiment of the present invention, the tracking process of the non-fixed object can be performed more easily.

  Further, according to the embodiment of the present invention, by calculating the image scale of the object on the imaging plane based on depth information and approximate triangle theory, more accurate object display and feature extraction can be easily performed, and more stable tracking can be performed. It becomes possible.

  In addition, a series of operations in this specification can be executed from hardware, software, or a combination of hardware and software. When executing the series of operations by software, a computer program may be installed in a memory of a computer built in dedicated hardware, and the computer program may be executed by the computer. Alternatively, a computer program may be installed in a general-purpose computer that can execute various processes, and the computer program may be executed by the computer.

  For example, the computer program is stored in advance on a recording medium such as a hard disk or ROM, or the computer program is temporarily or permanently stored on a floppy disk, CD-ROM, MO, DVD, magnetic disk, semiconductor memory, etc. It can be stored (recorded) on a mobile recording medium, and such a mobile recording medium may be provided as a package.

  Although the present invention has been described in detail with the specific embodiments described above, it goes without saying that modifications and substitutions can be made to the embodiments without departing from the spirit of the present invention. In other words, the present invention is disclosed in the form of description and should not be construed as limiting. The spirit of the invention should be determined from the appended claims.

Claims (10)

A communication area acquisition step of performing a three-dimensional communication area analysis on the input initial depth image and acquiring a communication area list of the initial depth image;
An initial object determining step for determining a communication area where the object is located from a known current position of the object in the initial depth image, and determining n feature points (n is a natural number) in an image portion corresponding to the communication area; ,
A three-dimensional communication area analysis of the subsequent depth image input after the initial depth image is performed, and the object communication area where the object is located is identified from each candidate communication area obtained from the communication area list of the subsequent depth image. A tracking step;
Wherein from said object connected domain identified in tracking step tracks the n feature points, have a, and object positioning step of updating the current position of the object,
The object tracking method , wherein a mutual interval between the n feature points is equal to or greater than a first predetermined threshold, and each of the n feature points is distinguished from a corner of an adjacent pixel point by a specific tracking operator . The object tracking method according to claim 1 , wherein the corresponding image portion from which the n feature points are extracted is located in a depth image or a color image synchronized with the depth image.   In the tracking step, the candidate communication area having the highest similarity with the result of adding the state change after motion prediction to the previously determined position of the object from each candidate communication area of the communication area list of the subsequent depth image The object tracking method according to claim 1, wherein an object communication area where the object is currently located is searched.   The object tracking method according to claim 1, further comprising an image scale determining step of determining an image scale of the object in the initial depth image of the object communication area identified in the tracking step after the tracking step. In the image scale determination step,
The average depth of the object communication area of the object location identified in the tracking step is dn, the average depth of the object communication area of the object location determined from the initial depth image is d0, and the object in the initial depth image 5. The object tracking method according to claim 4 , wherein the image scale is calculated by Sn = d0 / dn * S0 where S0 is an image scale and Sn is an image scale in the subsequent depth image of the object. In the object positioning step,
A feature point tracking step of tracking the n feature points by the KLT tracker from the subsequent depth image;
A class quantization step for performing class quantization of each tracked feature point from the mask image information of the object communication area and weighting each tracked feature point;
The object tracking method according to claim 1, further comprising: a grouping step of performing grouping of the tracking feature points, calculating a center point of the group, and updating a current position of the object. After the feature point tracking step, before the class quantization step, the tracking feature point further includes a first removal step for removing a feature point whose error during tracking of the feature point exceeds a second predetermined threshold. And
After the class quantization step and before the grouping step, the tracking feature points further include a second removal step of removing a predetermined proportional number of feature points having the largest distance from the center of gravity of the tracking feature points. The object tracking method according to claim 6 . After the grouping step, so that the total number of feature points of the object is n, and so that the mutual interval between the n feature points that are located in the object and are replenished is equal to or greater than a first predetermined threshold value. The object tracking method according to claim 6 , further comprising a replenishment step of replenishing new feature points. A communication area acquisition device that performs a three-dimensional communication area analysis on the input initial depth image and acquires a communication area list of the initial depth image;
An initial object determination device for determining a communication area of the object location from a known current position of the object in the initial depth image and determining n feature points (n is a natural number) in an image portion corresponding to the communication area; ,
A three-dimensional communication area analysis of the subsequent depth image input after the initial depth image is performed, and the object communication area where the object is located is identified from each candidate communication area obtained from the communication area list of the subsequent depth image. A tracking device;
Wherein from said object connected domain identified in tracking device to track the n feature points, have a, an object positioning device that updates the current position of the object,
An object tracking device , wherein the n feature points have a mutual interval equal to or greater than a first predetermined threshold, and each of the n feature points is distinguished from a corner of an adjacent pixel point by a specific tracking operator .   The object tracking device according to claim 9, wherein the corresponding image portion from which the n feature points are extracted is located in a depth image or a color image synchronized with the depth image.

Images