JP6525453B2 - Object position estimation system and program thereof - Google Patents

Description

  The present invention relates to an object position estimation system capable of tracking a specific object in an image by parallel processing of images from a plurality of cameras, and a program thereof.

  An object position estimation system capable of tracking a specific object in a video can analyze sports video, and can be used for services such as automatic umpire, broadcast of sports programs, sports data generation / distribution, and coaching. In addition, it can be used for various services such as a security system based on surveillance camera image analysis.

  In recent years, with the improvement of pattern recognition technology and computer speed, the performance of technology for detecting and tracking a specific object in a video has been enhanced. The development of this video analysis technology is particularly remarkable in sports scene analysis, and for example, various applications using a camera as a sensor have been proposed. The Hawkeye system of tennis, which is also used in Wimbledon, three-dimensionally tracks a tennis ball using a plurality of fixed camera images as sensors, and performs IN / OUT judgment involving a judge. Also, in 2014 FIFA World Cup, called the goal line technology, it analyzes the images of several fixed cameras and automates the determination of goals. Furthermore, advancements in real-time video analysis technology in sports, such as the TRACAB system that installs a large number of stereo cameras in a football stadium and tracks all players in the field in real time, are advancing.

  As an example of the real-time video analysis technology, in order to reduce the processing load for extracting the subject area and to prevent the extraction of the area other than the subject area prior to the area detection of the soccer ball, the foreground and background of the image There is disclosed a technique for effectively separating the two (see, for example, Patent Document 1).

  On the other hand, there is disclosed a technique for projective transformation of a ball object arranged in the real world space so that it can be recognized and traced by a camera connected to a tablet terminal, and the tracking result is reflected in the operation of an application in the tablet terminal (for example, , Patent Document 2).

  Also, as a person tracking technique using a plurality of cameras, a technique is disclosed for reducing the processing load using the recognition processing result of another camera with respect to the recognition processing result of a certain camera, and displaying the marked person as a mark (for example, Patent Document 3).

Unexamined-Japanese-Patent No. 2012-9980 Japanese Patent Application Publication No. 2013-532337 JP, 2006-229465, A

  Techniques such as three-dimensional tracking of tennis balls as described above and the TRACAB system that tracks all players in the field in real time, such as a predictable tennis ball and a soccer ball from a video taken only near the goal It is intended for soccer players, etc. who are slow in moving speed and relatively easy to track. However, a soccer ball at an arbitrary position in the field is difficult to predict its trajectory, travels at a high speed, and there is a high frequency of occurrence of occlusion (metaphor), so a technique for stably tracking such a soccer ball is It has not been established yet.

  In particular, the technique of Patent Document 1 is configured to effectively separate the foreground and the background of the image from the image features in order to extract the subject region in the video, but the image features are handled in soccer ball region detection. However, it is difficult to stably track a soccer ball which moves at high speed and has a high frequency of occurrence of occlusion (metaphor) without stable recognition processing such as machine learning.

  In the technique of Patent Document 2, projective transformation is also used for space recognition in the real world, but the image after projective transformation can be freely processed so as to facilitate detection of an object of interest. It is difficult to apply to the application of tracking a ball in real time to an image taken at an arbitrary position, such as a soccer stadium, at a wide position.

  Further, in the technique of Patent Document 3, the processing load can be reduced by using the recognition processing result of another camera with respect to the recognition processing result of a certain camera as a person tracking technique by a plurality of cameras. Objects that are not tracked in the same space and move in high-speed in tracking an object in real time with respect to an image taken at an arbitrary position such as a football stadium, etc. It is difficult to keep track of

  Therefore, with the above-mentioned existing techniques, it is not possible to predict the trajectory due to bounce or bounce from a player, occlusion is likely to occur in the player's shadow, and it is difficult to track such as moving fast when shooting. It has not been possible to estimate and track the position of such specific objects.

  SUMMARY OF THE INVENTION The object of the present invention is made in view of the above problems, and an object position estimation system capable of tracking a specific object in an image by parallel processing of an image by a plurality of cameras, and a program thereof It is to provide.

  In order to solve the above problems, the object position estimation system of the present invention performs tracking on individual camera images in order to parallel process images from a plurality of cameras to enable tracking of specific objects in the images. By automatically detecting the object region of interest and performing stable object position estimation processing using machine learning, and integrating the object position estimation processing results of each camera image to estimate the position of the object in the real space again, Implement more stabilized object position estimation. The object position estimation system of the present invention configured as described above performs projective transformation of estimation processing for each camera image into the same space, and estimates the final object position, and complements estimation processing for each camera image. By using it, tracking object can be traced with high accuracy.

  For example, in the object position estimation system of the present invention, stable ball tracking is enabled by individually detecting and tracking the ball area from a plurality of fixed camera images installed in a stadium and integrating the tracking results. The theory of machine learning is used to detect balls, allowing accurate ball detection with reduced effects of noise, and if the balls are hidden behind a player's shadows, even if they would normally become untrackable, Processing with multiple cameras and integration with the tracking status of each camera enables stable ball tracking. In particular, in the integration of the tracking results, each camera image is placed on the same image space viewed from directly above the field by projective transformation, and a robust ball estimation process is realized in consideration of the player's dense area. The present invention is particularly effective for estimating the position of a soccer ball, but is a technology applicable to a wide range of ball games such as tennis, volleyball, basketball, and table tennis other than soccer. Moreover, it is also possible to estimate the position of various objects, not only the ball, but the tracking object.

  That is, the object position estimation system according to the present invention is an object position estimation system capable of tracking a specific object in an image by parallel processing of images by a plurality of cameras, and an image which is continuous in frame units from each camera To generate an inter-frame difference image, and object candidate region extraction means for determining object candidate regions in the current frame by low-level features based on regions divided by the difference image, and the object candidates obtained from each camera Object candidate area recognition means for specifying an area assumed to include an object to be tracked among the areas by a high level feature based on machine learning, and an object candidate area specified by the high level feature obtained from each camera , Ball candidate domain associated between frames Among the trajectories of the object candidate area obtained from each camera, the trajectory having the highest value based on the index previously determined for each camera is selected as the trajectory of the object to be tracked among the trajectories of the object candidate region obtained from each camera Object trajectory selection means, projection conversion means for converting the image of the current frame obtained from each camera and the image coordinates of the trajectory of the object candidate area selected for each camera into a predetermined projective transformation image and integrating them; The projective transformation image is divided into blocks, and within each block, feature amounts represented by objects other than the object to be tracked, which are predetermined to be highly correlated with the object to be tracked, are calculated and compared. Object candidates located in unremoved blocks after excluding blocks whose feature amount is less than a predetermined value The apparatus is characterized by comprising block determination means for selecting a mark, and object position estimation means for determining, as a final ball position coordinate, ball candidate coordinates continuing for a predetermined time or more with respect to the selected object candidate coordinates. Do.

  Further, in the object position estimation system of the present invention, the object candidate area recognition means uses a high-level feature based on machine learning to recognize a ball candidate area recognized by a dual feature of a color histogram feature and a local binary pattern feature. It is characterized by specifying.

  Further, in the object position estimation system according to the present invention, the object trajectory selection means may use one of the velocity, the movement amount, and the tracking time of the object candidate area on the projective transformation image as an index previously determined for each camera. It is characterized by selecting the above as an index.

  Further, in the object position estimation system according to the present invention, the block determination unit targets a person as having high correlation with the object to be tracked, and sets the feature amount represented by an object other than the object to be tracked. It is characterized in that the occupancy rate of a person is targeted.

  Further, in the object position estimation system according to the present invention, the object position estimation means is configured to continue the ball for a predetermined time or more based on an elapsed time from a tracking end frame of each trajectory for the selected object candidate coordinates. The candidate coordinates are determined as final ball position coordinates.

Furthermore, the present invention, the object candidate area extracting means with the object position estimation system according to the present invention, said object candidate region recognizing means, the object trajectory generation unit, and in order to implement the functions of the object trajectory selection means To the computer for individually selecting the locus of the object candidate area relating to the specific object from the image of the camera to be processed in the parallel processing, a continuous image is input in units of frames from the camera to be processed to obtain an interframe difference image. Generating and determining an object candidate area in the current frame by a low-level feature based on the area divided by the difference image, and including an object to be tracked among the object candidate areas obtained from the camera to be processed Mechanics the assumed area Identifying with a high-level feature based on H., generating a trajectory of a ball candidate region associated between frames for an object candidate region identified with the high-level feature obtained from the camera to be processed; Selecting a locus having the highest value based on a predetermined index for each camera among the loci of the object candidate area obtained from the camera as the locus of the object to be traced by the program ;
The present invention relates to a specific object extracted from images of a plurality of cameras in order to realize the functions of the projective transformation means, the block determination means, and the object position estimation means in the object position estimation system according to the present invention. An image of the current frame obtained from each camera through each step executed by the program and an object candidate selected for each camera on a computer capable of tracking the specific object by integrating the object candidate area Converting the image coordinates of the locus of the area into a predetermined projective transformation image and integrating the projective transformation image, and dividing the projection transformation image into blocks, which are predetermined as having high correlation with the object to be tracked in each block By objects other than the object being tracked Calculating and comparing the feature quantities to be selected, excluding blocks whose feature quantity is equal to or less than a predetermined value, and selecting object candidate coordinates located in the non-excluded block; and predetermined time for the selected object And determining a continuing ball candidate coordinate as a final ball position coordinate.

  According to the present invention, an object position in real space can be estimated with high accuracy.

It is a block diagram showing an example of composition of an object position estimating system of one embodiment by the present invention. It is a flowchart which shows an example of operation | movement of the object position estimation system of one Embodiment by this invention. It is explanatory drawing of the candidate area | region extraction process of the object in the object position estimation system of one Embodiment by this invention. (A), (B) is explanatory drawing of the candidate recognition process of the object by the low level feature in the object position estimation system of one Embodiment by this invention. (A), (B) is a figure which illustrates the positive example and the negative example for machine learning used for the candidate recognition processing of the object by the high level feature in the object position estimation system of one embodiment by the present invention, respectively. It is a figure which illustrates the color histogram used for the candidate recognition process of the object by the high level feature in the object position estimation system of one Embodiment by this invention. It is explanatory drawing regarding the process result of the locus | trajectory production | generation process of an object in the object position estimation system of one Embodiment by this invention, and a locus | trajectory selection process. (A), (B) is explanatory drawing regarding the projective transformation process in the object position estimation system of one Embodiment by this invention. It is a figure which illustrates the projective transformation image in the object position estimation system of one Embodiment by this invention. It is explanatory drawing regarding the block determination process of the object in the object position estimation system of one Embodiment by this invention. It is explanatory drawing regarding the position estimation process of the object in the object position estimation system of one Embodiment by this invention. It is a figure which contrasts the identification precision and processing time of the high level feature in the object position estimation system of one Embodiment by this invention. It is a figure which contrasts the estimation error in the object position estimation system of one Embodiment by this invention. (A), (B) is explanatory drawing regarding the fitting probability with respect to the area | region radius of the presumed position of the object in the object position estimation system of one Embodiment by this invention.

(Configuration of object position estimation system)
An object position estimation system 1 according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an object position estimation system 1 according to an embodiment of the present invention.

  The object position estimation system 1 illustrated in FIG. 1 inputs a plurality of target object detection / tracking devices 2 each of which inputs individual camera images and independently processes them in parallel to estimate the individual object positions, and the plurality of target objects A target object position integration device 3 which integrates the processing results of the detection / tracking device 2 to estimate the object position. The target object detection / tracking device 2 includes an object candidate area extraction unit 21, an object candidate area recognition unit 22, an object trajectory generation unit 23, and an object trajectory selection unit 24. Further, the target object position integration device 3 includes a projection transformation unit 31, a block determination unit 32, and an object position estimation unit 33.

  The plurality of target object detection / tracking devices 2 are respectively configured in the same manner, and configured to input and parallel process individual camera images in real time. In this example, camera images obtained from four cameras (first to fourth cameras) (in particular, a frame image obtained from each camera is referred to as a 2D (dimensional) image) are subject object detection and tracking. Each target object detection / tracking device 2 is input to the device 2, estimates the object position in each 2D image, and outputs the processing result to the target object position integration device 3. Therefore, the target object position integration device 3 is configured to synchronize the processing results input from the plurality of target object detection / tracking devices 2 with the camera images of the cameras and estimate the final object position. .

  The plurality of target object detection / tracking devices 2 and the target object position integration device 3 can be realized by personal computers. For example, the target object detection / tracking device 2 and the target object position integration device 3 of the present embodiment can be realized by a computer with respect to the function of each component, and each component according to the present invention The program for realizing the above is stored in a memory (not shown) provided inside or outside the computer. By controlling a central processing unit (CPU) or the like included in a computer, a program in which processing content for realizing the function of each component is described is read from memory and executed as appropriate. The functions of the components of the target object detection / tracking device 2 and the target object position integration device 3 can be realized by a computer. Here, the function of each component may be realized by part of hardware.

  In the following, camera images from four cameras (first to fourth cameras) disposed at a soccer stadium are analyzed, and a soccer ball (hereinafter, also simply referred to as a “ball”) is regarded as an object to be tracked. An example will be described. The more the number of cameras, the better, but if there are a plurality of cameras, the object position estimation system 1 according to the present invention can be configured.

  First, the functions of the object candidate area extraction unit 21, the object candidate area recognition unit 22, the object trajectory generation unit 23, and the object trajectory selection unit 24 included in the target object detection / tracking device 2 will be described.

  The object candidate area extraction unit 21 receives a continuous image (2D image) from the corresponding camera in frame units to generate an inter-frame difference image, and the current frame (hereinafter referred to as "the current 2D image") is generated. It has a function of executing a process (object candidate area extraction process) of determining an object candidate area (in this example, a ball candidate area) based on low-level features based on the areas divided by the difference image.

  The object candidate area recognition unit 22 is a process of specifying an area assumed to include an object to be tracked by a high-level feature based on machine learning for the ball candidate area in the current 2D image determined by the object candidate area extraction unit 21 It has a function of executing (object candidate area recognition processing). In particular, the object candidate area recognition unit 22 includes an object to be tracked by extracting a ball candidate area recognized by a double feature of a color histogram feature and a local binary pattern (LBP) feature. Identify the area assumed as

  The object trajectory generation unit 23 refers to the ball candidate area identified by the object candidate area recognition unit 22 between frames (that is, one or more past 2D images continuous with the current 2D image (hereinafter referred to as “past 2D image”) ) And the function of generating the trajectory of the ball candidate area associated with the object trajectory generation processing).

  The object trajectory selection unit 24 selects a trajectory having the highest value based on a predetermined index described later among the trajectories of the ball candidate regions that can be generated in plurality by the object trajectory generation unit 23 and selects the trajectory together with the current 2D image. The image coordinates (hereinafter also referred to as “ball candidate coordinates”) of the ball candidate area in each frame of the determined trajectory is determined as the trajectory of the object to be tracked (in this example, the ball). It has a function of executing a process to output (object trajectory selection process).

  Next, the functions of the projection conversion unit 31, the block determination unit 32, and the object position estimation unit 33 included in the target object position integration device 3 will be described.

  The projective transformation unit 31 receives a 2D image (that is, a current 2D image) and ball candidate coordinates of the selected trajectory from each target object detection / tracking device 2 and performs predetermined projection reflecting the ball candidate coordinates. It has a function of executing processing (projective conversion processing) for converting and integrating each into a converted image.

  The block determination unit 32 divides the projective transformation image obtained by the projective transformation unit 31 into blocks according to a predetermined number of blocks, and displays a table by an object other than the ball which is predetermined as having high correlation with the object of the ball in each block. Each of the ball candidates obtained from the projective transformation image by calculating and comparing the feature quantities to be calculated, excluding blocks having the feature quantity equal to or less than a predetermined value, and selecting ball candidate coordinates located in the non-excluded blocks. It has a function of executing a process of selecting coordinates (block determination process). For example, the occupancy rate of a person area can be used as a feature quantity represented by an object other than the ball in each block. That is, as block determination processing, the occupancy rate of the person area of each block obtained from the projective transformation image is calculated, and based on the occupancy rate of the person area, ball candidate coordinates in blocks having high occupancy rate of the person area are prioritized Select to

  The object position estimation unit 33 continues, for each ball candidate coordinate selected by the block determination unit 32, for a predetermined time or more (a predetermined number of frames or more) based on the "time elapsed from the tracking end frame" of each trajectory. It has a function of executing processing (ball position estimation processing) of determining the ball candidate coordinates that are being played as the final ball position coordinates. The determined final ball position coordinates are displayed on a predetermined image display unit (not shown) in a manner indicating the position coordinates in the projective transformation image, and the display is continuously performed to estimate the final position. The trajectory of the ball position can be displayed on the projective transformation image. Then, by determining ball candidate coordinates continuing for a predetermined time or more (predetermined number of frames or more) as final ball position coordinates, for example, each ball candidate coordinate selected by the block determination unit 32 is present Even in the case where there are a plurality of frames or no frame is detected at all, when one frame is continuously displayed on each of the projective transformation images continuously projectively transformed in a plurality of frames, approximately one estimated ball position trajectory is recognized.

(Operation example of object position estimation system)
Hereinafter, an example of the operation of the object position estimation system 1 according to an embodiment of the present invention will be described in more detail with reference to FIGS. Although each drawing showing a 2D image related to a camera image is illustrated in gray scale, it is assumed to be processed with RGB color information unless otherwise specified. FIG. 2 is a flow chart showing an example of the operation of the object position estimation system according to an embodiment of the present invention.

  First, each of the target object detection / tracking device 2 inputs continuous video (2D image) in frame units from the corresponding camera (step S1).

  Next, each of the target object detection / tracking device 2 generates an inter-frame difference image for continuous video (2D image) in frame units input from the corresponding camera by the object candidate area extraction unit 21 and the current 2D The ball candidate area in the image is determined by the low level feature based on the area divided by the difference image (step S2).

  More specifically, the object candidate area extraction unit 21 generates an inter-frame difference image, and roughly extracts ball candidate areas based on low-level image features. The inter-frame difference image is an image in which the difference value for each pixel of an adjacent video frame is used as a pixel value, and the value increases only in a portion where motion occurs in the image. Since the ball at the time of tracking always moves, the ball can be efficiently detected by processing only pixels having a high value (absolute value) in the inter-frame difference image. Then, the inter-frame difference image binarized with a specific threshold value is subjected to labeling processing, and low-level features are calculated for each label (connected area). The low-level feature is a color, a shape (degree of circularity), and the number of pixels in the label, and a label having a feature amount outside the preset range is excluded from the ball candidate to determine the ball candidate region.

  More specifically, with reference to FIG. 3 and FIG. 4, FIG. 3 is an explanatory view of object candidate area extraction processing by the object candidate area extraction unit 21 in the object position estimation system 1 of one embodiment according to the present invention. is there. FIGS. 4A and 4B are explanatory diagrams of object candidate recognition processing by the object candidate area extraction unit 21 based on low-level features in the object position estimation system according to the embodiment of the present invention. For example, as shown in FIG. 3, it is possible to generate an inter-frame difference image by taking the difference between the n-th frame and the (n-1) -th frame including the moving object Ob. In the inter-frame difference image, labels L1 and L2 are attached to the areas of pixels having high values (absolute values). It is to be noted that the inter-frame difference image may be configured to be labeled after being subjected to filter processing by monomorphic calculation.

  In this manner, in FIG. 4 (A) exemplifying the n-th frame (the current 2D image Fr), labels L1 to L19 can be attached to the respective regions extracted as shown in FIG. 4 (B). The ball candidate area can be determined by calculating low-level features for each label (connected area) and excluding labels having feature quantities outside the preset range from the ball candidates. In FIG. 4B, the areas (labels) L1, L4 to L8, L12 to L17 excluded from the ball candidate area and the areas (labels) L2, L3, L9 to L11, L18, extracted as ball candidate areas. It categorizes into L19 and illustrates that the ball candidate area can be determined. The label 1 is regarded as one area when the areas extracted in the inter-frame difference image are dense in a predetermined amount or more, and can be excluded from the ball candidate areas.

  Next, each of the target object detection / tracking device 2 is assumed to include an object to be tracked for the ball candidate region in the current 2D image determined by the object candidate region extraction unit 21 by the object candidate region recognition unit 22 Region is identified by high-level features based on machine learning (step S3). In particular, the object candidate area recognition unit 22 is assumed to include an object to be tracked by extracting a ball candidate area recognized by a double feature of a color histogram feature and a local binary pattern (LBP) feature. Identify the area.

  More specifically, the object candidate area recognition unit 22 narrows down ball candidates with higher accuracy. To this end, it is preferable to use a supervised learning framework for ball recognition. First, a large number of ball images (12 × 12 pixels) and non-ball images (12 × 12 pixels) are collected in advance from a soccer video. FIGS. 5A and 5B respectively show a positive example (ball image) and a negative example (ball image) for machine learning used in object candidate recognition processing by high-level features in the object position estimation system 1 according to an embodiment of the present invention. It is a figure which illustrates a (non-ball image). Color histogram feature quantities and LBP feature quantities are extracted from the collected positive and negative example images.

  FIG. 6 is a view exemplifying a color histogram used for object candidate recognition processing by high-level features in the object position estimation system 1 according to an embodiment of the present invention. As shown in FIG. 6, the color histogram feature value can be expressed by dividing the RGB color space into 27 dimensions of 3 × 3 × 3 and by the color frequency of each pixel in the positive and negative example images.

  The LBP feature is a pattern-based feature used in texture analysis, and has recently been attracting attention because of high identification accuracy and low calculation cost (for example, “Terajima, Kida,” using gradient information) "Improvement of Local Binary Pattern", DEIM Forum 2014, F5-4). The LBP feature value is calculated by raster scan, and can be determined based on Equation (1), where R is the radius of the pixel of interest and P is the number of pixels in the vicinity region.

Here, g _c represents the pixel value of the target pixel, and g _p represents the pixel value of the reference point. When R = 1, the near region is 3 × 3 and the maximum of P is 8. The patterns LBP _{P and R} are calculated by comparing the size of the pixel of interest g _c with the pixel value g _p of the neighboring region, and the type of L BP feature amount is _P <2>. The LBP feature is a feature of a histogram describing the frequency of this pattern. As an example, a 256-dimensional LBP of R = 1 and P = 8 can be used. The total number of dimensions is 283 (= 27 + 256) due to the dual feature amount of the color histogram feature amount and the LBP feature amount. With this 283-dimensional feature quantity, the recognition processing load by machine learning can be relatively reduced, and real-time processing on a 2D image input in real time can be realized.

  Then, as this machine learning, a binary support vector machine (SVM: Support Vector Machine) is configured in a supervised learning framework using high-level features by the 283-dimensional dual feature amount. By using the classifier based on this support vector machine, the ball candidate area is judged with higher accuracy than the extraction based on the low level feature described above, and the ball is processed in real time with the 2D image input in real time. It is possible to narrow down the number of ball candidate areas assumed to include.

  That is, since the identification process by the support vector machine requires processing time, in the object position estimation system 1 according to the present invention, after the number of areas is reduced by the ball area extraction by the low level feature, the recognition process by the high level feature is performed. The high-precision and real-time processing is realized by adopting a configuration in which the two-stage extraction is performed to perform the

  Next, each of the target object detection / tracking devices 2 performs the current 2D image and one or more consecutive past 2D images for the ball candidate area identified by the object candidate area recognition unit 22 by the object trajectory generation unit 23. A trajectory of a ball candidate area associated with each other (that is, between adjacent frames) is generated (step S4).

  More specifically, for the ball candidate area extracted as the low-level feature and the high-level feature by the object-trajectory generating unit 23, a movement locus obtained by connecting the position coordinates of a predetermined number (for example, three points) or more on the time axis. Generate For example, it is assumed that there are m ball candidate areas extracted in frame i of the current 2D image and n ball candidate areas tracked by frame i-1 in the past 2D image. Under the present circumstances, it is based on the distance to each locus and the similarity of a low level feature to which locus of the ball candidate area currently traced by frame i to which a ball candidate area newly extracted by frame i is assigned. to decide.

  Then, it is assumed that low-level features of the ball candidate area tracked by frame i-1 are associated with each trajectory and held. There is a ball candidate area in which the ball candidate area extracted in frame i has a similarity higher than a predetermined value with respect to low-level features held in association with the trajectories, and the distance is short within a predetermined range. In the case, the ball candidate area is added to the trajectory as tracking success. On the other hand, if there is no ball candidate area whose similarity is higher than a predetermined value and the distance is short within the predetermined range, the ball candidate area extracted in the frame i is regarded as a new object, and the frame 2 of the current 2D image is considered. Generate a new trajectory from. Each trajectory generated by the object trajectory generation unit 23 holds image coordinates in a frame in which the ball candidate region has been successfully detected. In addition, trajectories for which tracking failed for a specified number of frames or more are deleted from the tracking targets.

  Next, each of the target object detection / tracking devices 2 causes the object trajectory selection unit 24 to generate a plurality of trajectory candidate ball candidates that can be generated by the object trajectory generation unit 23 based on the frame of the current 2D image. The trajectory having the highest value is selected based on a predetermined index for each trajectory composed of the above frames, and the ball candidate coordinates in each frame of the selected trajectory are selected along with the current 2D image, , And the ball) are determined (step S5).

  More specifically, the object trajectory selection unit 24 selects a trajectory having the highest value based on a predetermined index, and determines the final ball position in the current 2D image of the corresponding camera. Each trajectory generated by the object trajectory generation unit 23 holds image coordinates in a frame in which the ball candidate region has been successfully detected. These image coordinates can be converted to real space coordinates by projective transformation described later. The object trajectory selection unit 24 obtains, from the history of position coordinates of each trajectory, one or more of the velocity, movement amount, and tracking time of the ball candidate region in real space (that is, on the projective transformation image) as an index .

  For example, for each locus obtained on the basis of the frame of the current 2D image, the movement amount is calculated in frame units based on the image coordinates in each frame constituting each locus, and the movement amount calculated in frame units is calculated By averaging with the number of frames (tracking time) corresponding to the length of the locus, the evaluation value of the index (which corresponds to the average motion vector length) is obtained, and the one with the longest average motion vector length is Select as the trajectory closest to the movement of the ball. Alternatively, for each trajectory obtained based on the frame of the current 2D image, the velocity and movement amount are calculated in frame units based on the image coordinates in each frame constituting each trajectory, and the velocity calculated in frame units is calculated The evaluation value of the index can be obtained by averaging the number of frames (tracking time) corresponding to the length of each trajectory. Alternatively, it is possible to obtain an evaluation value of the index by a weighted addition of the average velocity and the average moving amount and the tracking time taken into consideration in the number of frames of each trajectory. Qualitatively, the one with the highest average velocity, the one with the highest average movement amount, and the one with the longest tracking time is selected as the trajectory closest to the movement of the ball. Then, the ball candidate coordinates in each frame of the selected trajectory together with the current 2D image are determined as final values of each of the target object detection / tracking device 2. Note that if the index is provided with a threshold and all the trajectories are below the threshold, that is, if it is determined that there is no trajectory close to the movement of the ball, tracking of the ball candidate area in the camera image is considered as failure. It considers and outputs NULL instead of the ball candidate coordinates in each frame of the selected trajectory.

  Furthermore, regardless of the selection of the trajectory by the object trajectory selection unit 24, the object trajectory generation unit 23 continuously tracks the trajectories of all the ball candidate areas when the plurality of ball candidate regions are generated. Therefore, the object trajectory selection unit 24 also temporarily cancels the trajectory of the ball candidate coordinates selected as the trajectory closest to the movement of the ball (that is, when the ball candidate coordinates assigned to the trajectory disappear), The image coordinates of the other ball candidate area closest to the distance in the frame at the interrupted time point are regarded as the ball candidate coordinates of the trajectory closest to the movement of the ball, and tracked and output up to the specified frame number. At this time, the object trajectory selection unit 24 adds “the elapsed time from the tracking end frame” at the time of the interruption as the auxiliary information, and outputs it along with the trajectory of the object to be tracked (that is, the ball). Each trajectory is always configured to have "elapsed time from the tracking end frame" as auxiliary information, and when it is possible to always track from the tracking start of the tracking target object to the current 2D image, "elapsed time from the tracking end frame "" Means "0".

  Normally, all the trajectories may lose ball candidate coordinates in consecutive frames, and even if the ball candidate coordinates are lost, the image coordinates of another ball candidate area closest in distance can be output by outputting the image coordinates. The trajectory of the ball candidate area finally selected by the object trajectory selection unit 24 can be traced and displayed up to the specified number of frames, and the tracking ability can be improved. Then, by adding “the elapsed time from the tracking end frame” at the time of the interruption as the auxiliary information, it is possible to improve the accuracy at the time of selecting the final ball position coordinate described later.

  An example of the locus selected by the object locus selection unit 24 is shown in FIG. FIG. 7 is an explanatory diagram of processing results of an object trajectory generation process and a trajectory selection process in the object position estimation system 1 according to the embodiment of the present invention. As the trajectory selected by the object trajectory selection unit 24, as shown in FIG. 7, the ball candidate coordinates Bn of the selected trajectory in the frame Fr of the current 2D image are the ball candidate coordinates for the past frames from the current 2D image Fr. Can be expressed as a locus TLn.

  Next, the target object position integration device 3 inputs the current 2D image of each and the ball candidate coordinates (or the NULL) in each frame of the selected trajectory from each of the target object detection / tracking device 2 ( Step S6).

  Next, in the target object position integration device 3, the 2D image (that is, the current 2D image) input from each target object detection / tracking device 2 by the projective transformation unit 31 and the ball candidate coordinates of the selected trajectory Are converted and integrated into predetermined projective transformation images reflecting the ball candidate coordinates (step S7).

  More specifically, the projective transformation unit 31 projective-transforms the 2D image obtained from each target object detection / tracking device 2 and the ball candidate coordinates of the selected trajectory into real space coordinates. Projective transformation is a transformation on an image represented by equations (2) and (3), which maps from plane to plane (for example, “experiment on geometric correction of Toei image,“ Takahashi, Numa, Aoki, Kondo ” Examination ”, the Tohoku Section of the Institute of Measurement and Control Engineers, No. 235, Conference No. 235-5, 2007.

Here, coordinates (x _c , y _c ) are coordinates before conversion, and coordinates (x _p , y _p ) are coordinates after conversion. The projective transformation in this example is a mapping from a plane to a plane and therefore becomes two-dimensional coordinates even after transformation, but since height can be omitted in two-dimensional field coordinates, the height component in real space is zero (z = 0) is considered. h ₁ ,..., h ₈ are projective transformation parameters, and the projective transformation matrix H is expressed by equation (4).

  The eight parameters in Equation (4) can be obtained if the correspondence between four or more points between images is obtained. For example, four or more characteristic points such as the four corners of the field are specified from the fixed captured video, and a projection transformation matrix is created in advance to transform the field into a video viewed from directly above. By calculating this projective transformation matrix, for example, a frame image Fr (n) (and ball candidate coordinates) of a certain camera shown in FIG. 8A can be displayed in the two-dimensional field coordinate system shown in FIG. It becomes possible to convert into an image IFr (and ball candidate coordinates after projective transformation). FIGS. 8A and 8B show an example of an image obtained by projective conversion of an image obtained by photographing the right side of the soccer field to the directly above coordinate system. Furthermore, it is illustrated, for example, in FIG. 9, for example, by combining an image obtained by project-transforming an image obtained by capturing the right side of the soccer field into the directly upper coordinate system and one obtained by project-transformed images obtained by capturing the left side of the soccer field. You can get an image. That is, the frame image obtained from each of the target object detection / tracking device 2 and the ball candidate coordinates of the selected trajectory are projective transformed into the same space (for example, field coordinates corresponding to the directly above image coordinate system) . The subsequent processing is performed in the projection coordinate system directly above.

  Next, in the target object position integration device 3, the block determination unit 32 divides the projective transformed image obtained by the projective transform unit 31 into blocks by a predetermined number of blocks, and has high correlation with the object of the ball in each block. By calculating and comparing feature quantities represented by objects other than the ball, which are predetermined as the above, excluding blocks whose feature quantities are equal to or less than a predetermined value, and selecting ball candidate coordinates located in unremoved blocks Each ball candidate coordinate obtained from the projective transformation image is selected. For example, the occupancy rate of a person area can be used as a feature quantity represented by an object other than the ball in each block. That is, as the block determination processing of the block determination unit 32, the occupancy rate of the person area of each block obtained from the projective transformation image is calculated, and based on the occupancy rate of the person area, the ball in the block having a high occupancy rate of the person area Select candidate coordinates with priority.

  More specifically, for example, as shown in FIG. 10, the block determination unit 32 divides the image IFr project-transformed as shown in FIG. 9 into 12 blocks, and the occupancy rate of the person area h in each block is calculate. An equation for calculating the occupancy rate is shown in equation (5).

O _i = m _i / p _i (5)

Here, O _i is the occupancy of the human region in the block number Bk, m _i is the total number of pixels in the person area, p _i is the total number of pixels in the block Bk. In general, balls are often located in blocks where players are closely packed. Therefore, when a ball is detected from a block having a low occupancy rate of a person area in a certain camera image, a stable ball position estimation can be realized by rejecting the detection. Note that the calculation of the feature amount (person area) represented by an object other than the ball in each block does not require strictness, and it can be determined based on the difference between the blocks. For example, arbitrary image analysis processing such as analysis using the color histogram described above can be used. For example, in FIG. 10, blocks other than Bk (0, 1), Bk (0, 2), Bk (1, 1), Bk (1, 2), Bk (2, 1), Bk (2, 2) Bk is excluded from the ball estimated position.

  Next, in the target object position integration device 3, for each of the ball candidate coordinates selected by the block determination unit 32 by the object position estimation unit 33, based on the “elapsed time from the tracking end frame” of each trajectory, a predetermined time The ball candidate coordinates continuing above (more than the predetermined number of frames) are determined as the final ball position coordinates (step S9). Each trajectory is always configured to have "elapsed time from the tracking end frame" as auxiliary information, and when it is possible to always track from the tracking start of the tracking target object to the current 2D image, "elapsed time from the tracking end frame "" Means "0".

More specifically, the object position estimation unit 33 integrates the ball candidate coordinates of each camera in the directly upper coordinate system, and determines the final ball position coordinates, so that the real space coordinates (two-dimensional field coordinates) are used. The final ball position coordinates P _F are calculated by weighted linear sum based on the elapsed time from the tracking end frame according to the equation (6) for the ball seat candidate coordinates of each camera video projective transformed.

Here, with frame number c as an element, t (c) is an elapsed time from the tracking end frame, and P _F (c) is obtained by being selected from each target object detection / tracking device 2 via the block determination unit 32 The ball candidate coordinates in question, T is a fixed time (for example, 3 seconds) defined as the maximum value of “the elapsed time t (c) from the tracking end frame”. An example of ball position estimation on an image of real space coordinates (two-dimensional field coordinates) is shown in FIG. Circles in the figure are the ball position coordinate points finally obtained by estimation, and black dots represent the coordinate points of the actual ball position. Also, although the positions of the ball candidate coordinates obtained in the processing for each of the four camera images preferably indicate the same portion, as shown in FIG. 11, the positions of the ball candidate coordinates at a plurality of different portions Is also assumed, and is indicated by a number representing each camera number. In FIG. 11, the camera number 2 indicates that the elapsed time t (c) from the tracking end frame is the longest, and according to Equation (6), “T−t (c)” is obtained by the camera. The ball candidate coordinates having a longer elapsed time t (c) from the tracking end frame are displayed longer as the final ball position coordinates P _F.

  That is, the determined final ball position coordinates are displayed on a predetermined image display unit (not shown) in a manner of indicating the position coordinates in the projective transformation image, and the display is continuously performed, thereby the final The trajectory of the estimated ball position can be displayed on the projective transformation image. Then, based on "the elapsed time from the tracking end frame", ball candidate coordinates continuing for a predetermined time or more (for example, a predetermined number of frames such as 3 frames or more) are set as final ball position coordinates. decide. Thus, for example, even in the case where there are a plurality of ball candidate coordinates selected by the block determination unit 32 in a certain frame or none of them are detected at all, it is possible to continuously project transformed images in a plurality of frames. When continuously displayed, a track of approximately one estimated ball position is recognized.

Note that if trajectories of a plurality of ball candidate coordinates are recognized even after the processing of the block determination unit 32, the block with the highest priority (for example, the block with the highest occupancy rate of the person) In the above, the ball candidate coordinates having the longest “time elapsed from the tracking end frame” may be determined as the final ball position coordinates P _F.

  Also, a Kalman filter can be used as a ball position state estimation algorithm instead of the equation (6) based on the "time elapsed from the tracking end frame". The Kalman filter is used to estimate the time-varying amount (for example, the position and velocity of an object) from observation with discrete error (for example, “Nishiyama,“ Kalman filter ”, Institute of Electronics, Information and Communication Engineers,“ Knowledge base, 1 group (signals and systems)-5 (signal theory)-Chapter 6, chapter 2011). However, as shown in the experimental results described later, the state estimation equation defined as equation (6) is superior to the Kalman filter in real-time ball position estimation.

  An experimental result comparing and verifying the object position estimation system 1 of the present embodiment with the conventional technique will be described below.

(Experimental conditions)
In order to confirm the effectiveness of the object position estimation system 1 of the present embodiment, a verification experiment using a soccer match video was performed. At the 2014 Emperor's Cup All-Japan Soccer Championship, 4 camera images of the field fixedly shot were used for verification. We learned the video of the first half of each camera and used the video of the second half for evaluation. The ball position on the two-dimensional field image was visually checked every 10 frames and used as the correct data of the ball position.

(Ball detection accuracy)
First, the performance evaluation of the high-level feature extraction process of the object candidate area recognition unit 22 obtained through the low-level feature extraction of the object candidate area extraction unit 21 according to the present invention was performed. Using about 9,000 ball images (positive example) and non-ball images (negative example), half of them were used for learning and the other half were used for evaluation. The discrimination accuracy by binary determination and the processing time per image required for discrimination are shown in FIG. For comparison, moments, color histograms alone, LBP alone, ORB, SURF, and SIFT are shown. The “moment” in this experiment is one of the techniques for quantitatively expressing the features of images and shapes, and in this example, the zero-order spatial moment, central moment, normalized central moment, and translation and rotation The seven invariant Hu moment invariants are combined into 10 dimensional feature quantities. In the “color histogram” according to this experiment, the RGB color space is a 27 × 3 × 3 × 3 feature amount. “LBP” according to this experiment is a 256-dimensional feature quantity of R = 1, P = 8 based on the equation (1). “ORB”, “SURF”, and “SIFT” are conventionally known as features due to local gradients in an image, and identification was performed using a histogram with bins of the feature dimensions.

  The “present invention” shown in FIG. 12 according to the present experiment is 283 based on the double feature quantity of the color histogram feature quantity (27 dimensions) and the LBP feature quantity (256 dimensions) as extraction based on high level features of the object candidate area recognition unit 22. It is a feature of dimension. As shown in FIG. 12, the discrimination performance of the LBP and the color histogram is higher than that of the other techniques even if it is a single device, and the double feature amount combining the both as in the object position estimation system 1 of this embodiment is the best. It became a high precision value. It has been confirmed that the processing speed of the object position estimation system 1 of the present embodiment can be realized with a load within a range where real time processing can be realized although it is slightly slower than LBP alone and color histogram alone.

(Ball position estimation accuracy)
Subsequently, the estimation accuracy of the ball position by the object position estimation system 1 of the present embodiment was verified. The error between the correct position and the estimated position of the ball on the two-dimensional field coordinates was calculated from the video of the second half of the game. The result compared with the Kalman filter which is a typical object tracking technique is shown in FIG.

  When the estimation error was converted into meters in the real space, in the object position estimation system 1 of the present embodiment, the average value of the errors was 5.42 m. On the other hand, in the case of the Kalman filter, the average value of the errors is 10.55 m, and the object position estimation system 1 of this embodiment suppresses the error low compared to the Kalman filter, and can estimate the ball position with high accuracy. It could be confirmed that The probability that the estimation error falls within a circle of a predetermined radius is also considered as an important factor of identification accuracy. For example, in FIG. , Shows the case where it matches the actual ball position. Further, in FIG. 14B, the ball candidate coordinates of camera number 4 among the ball candidate coordinates of camera numbers 1 and 4 are estimated as final ball position coordinates, and a slight deviation from the actual ball position occurs. There is. The probability that this estimation error falls within a circle of a predetermined radius is also shown in FIG. 13, and it was confirmed that an error within a radius of 10 m could be achieved in about 90% of the time slot of the game.

  In the object position estimation system 1 of the present embodiment, in particular, it is possible to effectively select an object close to the movement of the ball from the locus, in that high precision ball detection processing using machine learning is performed on each camera image. And the point that the processing results for a plurality of camera images are complementarily integrated are considered to have contributed to the high accuracy.

  Although the present invention has been described above by giving an example of a specific embodiment, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the technical concept thereof. For example, in the object position estimation system 1 according to the present embodiment, an example in which the plurality of target object detection / tracking devices 2 and the target object position integration device 3 are configured as separate devices has been described. One of the tracking devices 2 may be configured to incorporate the function of the target object position integration device 3. Also, depending on the application, the functions of the plurality of target object detection / tracking devices 2 and the target object position integration device 3 may be incorporated into one device. Therefore, the plurality of target object detection / tracking devices 2 and the target object position integration device 3 can be realized by personal computers, can be configured to operate with individual programs, and can be implemented as a single program. It can also be configured.

  Also, in the above-described example, the block determination unit 32 has described, as an example, occupancy rates by a plurality of persons for other objects that can be predetermined as objects to be tracked that are highly correlated. If the number of athletes is limited, for example, each block may be configured to determine the proportion occupied by the athlete itself.

  In the above-described example, the projective transformation image is described as a two-dimensional field image, but any projective transformation image is formed as long as it can be defined from a three-dimensional space to a two-dimensional space. be able to.

  According to the present invention, it is possible to estimate the object position in real space with high accuracy, so it is used for sports video analysis technology, in particular, services such as automatic refereeing, sports program broadcasting, sports data generation / distribution, and coaching. It is possible. If the tracking target is expanded to a specific object, it can be applied to various services such as a security system based on surveillance camera image analysis. Therefore, it is useful for the application which requires estimation of the object position in real space with high accuracy.

DESCRIPTION OF SYMBOLS 1 object position estimation system 2 target object detection and tracking device 3 target object position integration device 21 object candidate region extraction unit 22 object candidate region recognition unit 23 object trajectory generation unit 24 object trajectory selection unit 31 projection conversion unit 32 block determination unit 33 object Position estimation unit

Claims (6)

An object position estimation system capable of processing an image by a plurality of cameras in parallel to enable tracking of a specific object in the image,
Object candidate area extraction means for inputting a continuous image in frame units from each camera to generate an inter-frame difference image, and determining object candidate areas in the current frame by low-level features based on areas divided by the difference image; ,
Object candidate area recognition means for specifying an area assumed to include an object to be tracked among the object candidate areas obtained from each camera by high-level feature based on machine learning,
Object trajectory generating means for generating a trajectory of a ball candidate region associated between frames for an object candidate region specified by the high level feature obtained from each camera;
Object trajectory selection means for selecting, as a trajectory of an object to be tracked, a trajectory having the highest value based on a predetermined index for each camera among trajectories of the object candidate area obtained from each camera;
Projection transformation means for transforming the image of the current frame obtained from each camera and the image coordinates of the locus of the object candidate area selected for each camera into a predetermined projective transformation image and integrating them;
The projective transformation image is divided into blocks, and within each block, feature amounts represented by objects other than the object to be tracked, which are predetermined to be highly correlated with the object to be tracked, are calculated and compared. Block determination means for excluding blocks having a feature amount equal to or less than a predetermined value and selecting object candidate coordinates located in unremoved blocks;
Object position estimation means for determining, as final ball position coordinates, ball candidate coordinates continuing for a predetermined time or more with respect to the selected object candidate coordinates;
An object position estimation system comprising:   The object candidate area recognition means specifies a ball candidate area recognized by a dual feature quantity of a color histogram feature quantity and a local binary pattern feature quantity by a high level feature based on machine learning. Object position estimation system described in.   The object trajectory selection means is characterized in that one or more of the velocity, the movement amount, and the tracking time of the object candidate area on the projective transformation image is selected as an index determined in advance for each camera. The object position estimation system according to claim 1 or 2.   The block determination unit targets a person as having high correlation with the object to be tracked, and targets an occupancy rate of the person as a feature represented by an object other than the object to be tracked. The object position estimation system according to any one of claims 1 to 3, wherein   The object position estimation means determines, as the final ball position coordinates, ball candidate coordinates continuing for a predetermined time or more based on the elapsed time from the tracking end frame of each trajectory for the selected object candidate coordinates. The object position estimation system according to any one of claims 1 to 4, characterized in that: The object position estimation system according to claim 1, wherein the object candidate area extraction unit, the object candidate area recognition unit, the object trajectory generation unit, and the parallel processing of the object trajectory selection unit are realized. A computer individually selecting trajectories of an object candidate area relating to the specific object from images of a camera to be processed in
Generating a frame-to-frame difference image by inputting a continuous image in frame units from a camera to be processed, and determining an object candidate region in the current frame according to a low-level feature based on a region divided by the difference image;
Identifying a region assumed to include the object to be tracked among the object candidate regions obtained from the camera to be processed, using a high-level feature based on machine learning;
Generating a trajectory of a ball candidate area associated between frames for an object candidate area specified by the high level feature obtained from the camera to be processed;
Of trajectory of the object candidate area obtained from the processing target camera, comprising the steps of selecting a trajectory with the highest value on the basis of the predetermined metric for each camera as the locus of the tracking target object was executed by the program Above,
The object position estimation system according to claim 1, wherein the projection conversion means, the block determination means, and the object position estimation means are realized by the respective features extracted from the images of the plurality of cameras. On a computer that integrates object candidate areas related to objects and enables tracking of the specific object;
The step of integrating by converting the image of the current frame obtained from the camera through the steps executed by the program, the image coordinates of the trajectory of the object candidate region is selected for each camera to a predetermined projection transformation image When,
The projective transformation image is divided into blocks, and within each block, feature amounts represented by objects other than the object to be tracked, which are predetermined to be highly correlated with the object to be tracked, are calculated and compared. Removing blocks having a feature amount equal to or less than a predetermined value, and selecting object candidate coordinates located in unremoved blocks;
A program for causing a ball candidate coordinate continuing for a predetermined time or more with respect to the selected object candidate coordinate to be determined as a final ball position coordinate.