JP2002063577A - System and method for analyzing image, and image analysis program recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make outputable information on the states of stablly tracing objects and the union and separation of a plurality of objects by introducing a diversity such as one-to-multiple and multiple-to-one to the correspondence relation among objects between image frames. SOLUTION: The position and size of an object are acquired from an image area occupied by the object in respective image frames constituting an image, then, the transition cost in the case where the object in the former frame moves to the object in the latter is calculated about the set of the objects on the two frames in regard to the objects detected in the two adjacent frames, the correspondence relation between the objects on the two frames is classified from the transition cost, and the state of the objects is decided. The moving track of an individual object is traced over a plurality of image frames in the above correspondence relation of the objects by utilizing the information of results obtained by performing calculation about all adjacent inter-frames in the section of the image composed of a plurality of image frames.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image analysis system for tracking many objects displayed on a video,
Remote monitoring of traffic flow and human behavior using video obtained by video cameras, indexing of human motions in video in construction of video archives, analysis of athlete's movement status and team strategy from sports videos, etc. The present invention relates to an image analysis processing technique.

[0002]

2. Description of the Related Art Up to now, an image of a moving object such as a person or a vehicle has been input and the position of an object in the image has been detected and tracked over a plurality of frames.
Image processing techniques for recording the behavior of a moving object exist.

[0003] As a typical technique, the area, shape, color information, and the like of a target object are registered as a template at the stage of starting tracking, and template matching is performed in a subsequent image frame to determine the position of the best match. A moving object is tracked by calculating the position of an object and repeatedly executing this operation over a plurality of frames.

In the technique described in Japanese Patent No. 2697072, "Moving image object tracking device", the position of an object on an image frame on the next image frame is estimated to determine the correspondence. In the “automatic tracking device” of Japanese Patent No. 3026850, a color histogram of an object and its frame is calculated, a representative characteristic amount is determined by comparing them, and the position of the object in the next frame is searched by using the characteristic amount as a clue. ing.

[0005] These methods can exhibit good tracking performance when an object exists alone, its shape and color change is small, and its motion is simple and prediction is easy.

[0006]

However, when a plurality of persons are densely present or contact each other as in a soccer game, an area where a plurality of players overlap is generated in the video. The reason for the overlap is that people who are far apart in the three-dimensional space are projected on the two-dimensional image plane, so that they appear to overlap on the image, and in addition, the actual opponent's player's mark act It is conceivable that there is an overlap due to a plurality of persons touching in a three-dimensional space.

[0007] In such a video, in the conventional method, when a plurality of persons overlap, template matching cannot be performed stably and accurately, and the tracking information to be output becomes extremely inaccurate. There was a problem that would. In the conventional method, in order to deal with such a problem, when detecting a condition such as overlapping of persons, the tracking process is terminated, or the state shifts to a state of waiting for input for manual correction. Measures were taken. As described above, according to the conventional method, tracking processing cannot be stably performed on an image in which a plurality of people are crowded, and a state such as intersection or separation of a plurality of people cannot be recognized.

Accordingly, the present invention introduces diversity such as one-to-many, many-to-one, etc. into the correspondence between objects between image frames, and determines the state of movement, uniting, division, and disappearance of objects on an image. Accordingly, an object of the present invention is to provide a technology that enables stable tracking of an object and output of information on the state of merging and separation of a plurality of objects.

[0009]

SUMMARY OF THE INVENTION According to the present invention, an image obtained by photographing one or more objects is input, the object is tracked over a plurality of image frames constituting the image, and the object in each image frame is tracked. A multi-object tracking image analysis system for outputting information including the position of an image of a subject, comprising: means for extracting an image area occupied by an object in each image frame forming a video; The means for acquiring the center coordinate value and the size of the object, and the transition cost when the object in the previous frame moves to the object in the subsequent frame for the objects detected in two adjacent frames, Means for calculating a set of objects on two frames, and classifying the correspondence between the objects on the two frames based on the transition cost, and the appearance, disappearance, and Means for judging the state such as merging or splitting of a person, and the result of calculating the correspondence of an object between the two frames for all adjacent frames in a section of a video composed of a plurality of image frames Using the information of the object, the trajectory of each object is tracked over a plurality of image frames. In addition to the position and size of the object in each image frame, its appearance, disappearance, and Means for outputting the information including the information on the state of merging and division.

According to the embodiment of the present invention, as means for extracting an image area occupied by an object in an image frame, means for calculating a histogram for each color component of pixel values constituting an image, Means for calculating a range of color components in the background region of the object by calculating a color component value indicating the maximum value of the target object and a color component value corresponding to a valley adjacent to the peak having the maximum value as a peak. Means for calculating a ratio of a pixel value in a neighboring area centered on an arbitrary pixel in an image to a range of a color component of the background area; Means for calculating the ratio of the magnitude of the variance of the entire color component, and wherein the calculated variance ratio is equal to or less than a certain value, and wherein the pixel values in the neighboring region fall within the range of the color component of the background region. If the included ratio is above a certain value Determines that the pixel of interest is included in the area of the object, characterized by having a means for extracting a region of the object is a set of these pixels as binary image.

Further, according to the embodiment of the present invention, as means for acquiring the center coordinate value and the size of the object from the image area occupied by the object, an image holding the area occupied by the object as a binary image is used. Means for performing a distance change on the distance image to calculate a distance image, means for detecting a coordinate indicating a local distance value peak in the distance image, and an object centered on the coordinate indicating the distance value peak. Means for calculating an appropriate circumscribed rectangle for the area; outputting the center coordinates of the object as the center position of the circumscribed rectangle;
Means for calculating and outputting the size of the object as the height.

[0012] According to the embodiment of the present invention, for the objects detected in the two adjacent frames, the transition cost when the object in the previous frame moves to the object in the subsequent frame is reduced. As a calculation method, the distance between the objects on the two frames as the transition cost related to the size of the movement of the object, and the displacement of the entire image between the frames due to the camera's viewpoint movement in each of the horizontal and vertical directions Means for calculating by compensating and correcting the horizontal distance by the width of the object, and normalizing the vertical distance by the height of the object, and a transition relating to changes in the shape and color of the object As the cost, the spatial distribution of the pixel values of the object on one frame is superimposed on the object on the other frame, and the value that minimizes the error in the distribution of the pixel values is calculated. And having a means for.

According to an embodiment of the present invention, the correspondence between objects on two frames is classified by using the transition cost, and the appearance and disappearance of the objects and the union of the objects are combined. A means for determining the state such as split, split, etc., for each object on two frames, a means for determining that there is a correspondence between a set of objects whose transition cost is less than a certain value, The correspondence between the objects of
Single object in the previous frame corresponds to a single object in the rear frame, multiple objects in the previous frame correspond to a single object in the rear frame, single object in the previous frame An object corresponds to multiple objects in the rear frame, a state exists in the previous frame but no corresponding object exists in the rear frame, and an object exists in the rear frame but corresponds to the previous frame. Classify as non-existent state and simply move the single object in the previous frame to the single object in the subsequent frame. Move multiple objects in the previous frame to the single object in the subsequent frame. The moving state is the merging of objects, the single object in the previous frame is moving to multiple objects in the subsequent frame, the splitting of the object, the object existing in the previous frame but corresponding to the rear frame. Object disappears when there is no , Characterized by having a means for determining the state of an object is present on the rear frame corresponding to the previous frame does not exist as the appearance of the object.

Further, according to the embodiment of the present invention, the correspondence of the object between the two frames is calculated from the result of calculating between all the adjacent frames in the video section composed of a plurality of image frames. As a means of tracking the trajectory of each object over multiple image frames and outputting it including information on its appearance, disappearance, and merging and division of the objects over multiple frames Input the data of the existence of the object and the correspondence of the object between frames,
Means for storing the detected object in each image frame as a graph-structured data structure in which the vertices are the objects detected in each image frame and edges indicate the correspondence between the objects in adjacent frames; Means for searching the graph to obtain the movement trajectory of the object, and searching for the object having the state immediately after the division, until the state immediately before the disappearance or the united state is detected from the starting point. And a means for searching the graph until the state immediately before disappearance or the united state is detected from the start point, and acquiring the movement trajectory of the object.

In the case where a video image of a ball game played by a plurality of humans is input as the input video, for example, the background is a field area of a stadium, and the object is a player, a referee, etc. A person.

When photographing a person who exercises violently, such as in a soccer game, the shape of the person image in the video changes significantly from frame to frame. The tracking method using the conventional template matching has a problem that it is difficult to accurately track an object whose shape changes greatly between frames. However, in the present invention, the tracking method on two adjacent frames is difficult. Since the possibility of movement is calculated as the transition cost for all combinations of objects, there is an advantage that accurate tracking is possible even for an object having a large shape change of the object.

Also, in the case of a soccer game image or the like, since a plurality of persons collide or come into close contact with each other, a plurality of person images also appear densely in the image. It is extremely difficult to separate and separate individual person's areas from such dense areas.
Further, when another player is hidden behind a certain person, it is difficult to detect the hidden player from an image taken from a single viewpoint.

For the above reasons, in the present invention, when a plurality of persons are crowded or overlapped, a method of extracting the entire area of them as one person area is used, and the correspondence between the preceding and succeeding frames is used. Thus, by judging the combination or separation of a plurality of players, it is possible to cope with an overlap of a plurality of persons. On the other hand, in the conventional template matching method, a normal operation cannot be expected when a plurality of persons are crowded or overlapped.

In addition, when the conventional method is used for such a complicated object, it is necessary for a person to judge an overlapping situation of a person and to control a tracking process. It is possible to judge the union and division of the persons, and in that respect, it is superior to the conventional method.

Each section of the processing means described above can be realized by hardware, or can be realized by a software program executed by a computer.
The software program can be provided by being recorded on a compact disk, a floppy (registered trademark) disk, or another recording medium.

[0021]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a diagram for explaining an embodiment of the present invention.
0, processing unit 200, file system 300, memory 4
00, and a display unit 1000.

The input unit 100 includes a photographing device 101, a video image storage / playback device 102, and a video capturing device 103. The photographing device 101 is a camera that photographs a game video. The video image storage / playback device 102 is
1 is a device capable of recording and reproducing the video taken by the camera 1. Here, the photographing device 101 and the video image storage / playback device 102 need not be at the same point,
It is also possible to transmit a video image captured by the image capturing apparatus 101 through a communication line or the like, and receive and record the video image by the video image storage / reproduction apparatus 102. Further, the video image storage / reproduction device 102 does not need to be one, and it is also possible to use a plurality of devices to execute recording and reproduction by separate devices.

The video capturing device 103 is a device for capturing the video reproduced by the video video storage / reproducing device 102 on a computer and storing the video as digital data in the video storage file system 301. Video capture device 1
In step 03, when the output of the video image storage / reproduction device 102 is an analog signal, digital / analog (D / A) conversion is performed. Further, when the output of the photographing device 101 or the video image storage / reproduction device 102 is a digital signal, a function of converting the format of video data as necessary is also realized.

The processing unit 200 is a part for realizing the core technology of the present invention, and includes a histogram calculation unit 201, a field region extraction unit 202, a person region extraction unit 203, a person detection unit 204, a transition cost calculation unit 205, a frame It consists of an interval correspondence determination unit 206 and a person transition search unit 207. The processing unit 200 receives a video, detects a person from each image frame of the video, tracks the person, and as a result,
The position information of the person in each frame is output.
Details will be described later.

The file system 300 includes a video storage file system 301 and a person color model file system 3
02, a person detection result file system 303, and a tracking output file system 304.

The video storage file system 301 can store video data output from the video capture device 103 and output an image frame in an arbitrary video in response to a request from the processing unit 200. The person color model file system 302 is a file system in which color information of a person serving as a model used when determining the attribute of the person detected by the person detection unit 204 is stored.

The person detection result file system 303
This is a file system for storing data including the position and size of a person in each image frame of the input video output from the processing unit 200 and the correspondence between the person and the person on the preceding and succeeding frames. The tracking output file system 304 is a file in which the moving trajectory information of each person in the video output from the person transition search unit 207 is stored.
This is the final output of this system. Note that these file systems can be realized using the same or different fixed disk devices.

The memory 400 is an image frame memory 40
1, histogram memory 402, field area image frame memory 403, person area image frame memory 40
4, a person information memory 405, a transition cost matrix memory 406, a person state table memory 407, and a person transition structure memory 408. These memories are stored in the processing unit 2
00 is used to temporarily store input / output information and can be realized using a semiconductor memory.

The display section 1000 displays an image of the tracking result.

FIG. 2 is a flowchart showing a processing example of the system shown in FIG. First, an image frame is read from a video (Step 501), a histogram is calculated for each color component (Step 502), a field region is extracted from the image frame (Step 503), and a person region existing on the field is extracted. (Step 504),
A person is detected from the person area, its position and size information is obtained (step 505), and if the current image frame is not the start frame, the transition cost between the person and the person detected in the previous frame is calculated. (Step 50
6), the correspondence between the person in the current frame and the person in the previous frame is performed (step 507), and the result is output to a file,
Further, after the processing is completed up to the last frame, the person information obtained for each frame and the correspondence information are input (step 508), a person is searched using the information, and the result is output (step 508). 509).

Hereinafter, the operation of the processing section 200 shown in FIG. 1 will be specifically described. In the processing unit 200, first,
The video data stored in the video storage file system 301 is accessed, and image frames are read at every frame interval specified by the user in a frame section specified by the user, and stored in the image frame memory 401.

The histogram calculator 201 calculates a histogram (frequency distribution) for each color component for all pixels constituting the image frame stored in the image frame memory 401 and stores the histogram in the histogram memory 402.

An image handled by the present system is a color image, and each pixel holds a plurality of values in order to express a color. For example, a set of three numerical values of an RGB color system (red, green, and blue) is used in a video display device for a computer. The histogram calculation unit 201 can use the HSV color system (hue, saturation, luminance) as one of such color expression means.

Prior to the calculation of the histogram distribution, a division width of each color component value is set in advance, a portion between the minimum value and the maximum value of each color component is divided into a plurality of sections, and a memory capable of storing a frequency for each section. An area is secured in the histogram memory 402, and the value of each section is reset to 0.

Next, each pixel is circulated, and the value of the section of the histogram corresponding to the value of the color component is increased by one.
Here, an independent histogram is created for each color component. A histogram distribution is obtained from the values in the histogram memory after all the pixels have been visited. For example, a distribution as shown in FIG. 3A is obtained for each color component, and the highest mountain in the histogram is formed in a portion corresponding to the color of the field of the stadium.

Although the nature of the image changes significantly depending on the stadium and the lighting conditions at the time of shooting, the present invention can cope with images under various conditions by adjusting the division width of the histogram.

Although the HSV color system is used as an example of implementation in the histogram calculation unit 201 and subsequent processing, the present invention can be implemented using any other color system.

The field region extraction unit 202 extracts a field region in an image frame by using the image frame stored in the image frame memory 401 and the histogram distribution stored in the histogram memory 402 as inputs. The field area is output to the field area image frame memory 403 as an image. Here, the "field" does not refer to a field in the video interlaced system but refers to an area where a game is played in a stadium where the video was taken.

An image targeted by the present system generally has a configuration as shown in FIG. Reference numeral 601 denotes an image area other than the field area, which is an area corresponding to a stadium facility such as spectator seats; 602, a field area where a game is played; and 603, an image occupied by a person (player) existing on the field. Area.

In the field area extraction unit 202, first,
Using the histogram distribution, the image corresponding to the field
Calculate the range of elementary colors. Therefore, first, each color component
The maximum frequency for each histogram distribution
Value to the center color component (H_C, S _C, V_C)
You.

Next, the lower and upper limits of the color range of the field are determined for each component. In the present invention, the peaks sandwiching the peak of the histogram are defined as the color region of the field, and the color component values of the valleys sandwiching the peak are obtained by searching (FIG. 3).
(B)). For example, in the search for the lower limit, the gradient of the distribution is calculated while proceeding in the direction in which the color component value is lower than the peak value, and the color component value at which the gradient becomes zero can be obtained as the lower limit value. The upper limit is also obtained by reversing the search direction. The color range obtained here is determined for each component by (H _min , H _max ), (S _min , S _max ), (V _min ,
V _max ). The fact that the color component of a certain pixel is included in the field color area means that all the values of the hue H, the saturation S, and the luminance V are all included in the corresponding color range. In addition, this condition is relaxed, for example, the hue H and the saturation S
It is also possible to use various combinations of conditions, such as imposing conditions only on components.

Next, each pixel of the image frame is circulated to determine whether or not each pixel belongs to the field area.
The field area is smoother and has less color variation than other areas, so local color distribution variation and
By comparing the color variation of the entire field area, it is determined that a pixel having a locally uniform color distribution and conforming to the color range of the detected field is included in the field area.

As a specific method for this purpose, the pixel (i, j) in the image frame is circulated, and for a set of N × N pixels including the pixel of interest, a set of pixels included in the field color range is obtained. A ratio R (dispersion ratio) between the ratio r and the magnitude of the variance between the neighboring color distribution and the color distribution on the field is calculated. If the ratio r is equal to or greater than a certain value and the variance ratio R is equal to or less than a certain value, it is determined that the pixel belongs to the field. In that case, the corresponding coordinate value (i, j) on the field area image frame memory 403 becomes 1, and is set to 0 when it is determined to be out of the field.

In the field area determined as described above,
A region of a person (player) existing on the field may not be included and may remain as a hole in the field region. Therefore, the field area image frame memory 40
The shaping process is performed on the field region image stored in No.3. In the shaping process, first, isolated minute regions are removed as dust by a reduction process using a morphological filter. Next, in order to fill the hole in the person area, each pixel on the field area image is circulated, and when a certain pixel is 0 (set outside the field), as shown in FIG. The values of pixels at a certain distance are acquired, and when all of them are 1 (that is, on the field), the process of changing the value of the currently traveling pixel to 1 is repeatedly performed. FIGS. 5A, 5B, and 5C show patterns of neighboring pixels to be checked at that time, which are sequentially executed for one or a plurality of patterns. With this processing, it is possible to fill a large-scale hole in the area at a higher speed as compared with the expansion processing of the conventional morphological filter.

By the above processing, a field region as shown in FIG. 6 is obtained as a binary image on the field region image frame memory 403 from the image as shown in FIG.

The person area extraction unit 203 receives an image in the image frame memory 401 and a field area image in the field area image frame memory 403, extracts a person area from the image, and uses the extraction result as an image to extract a person area. The data is recorded in the frame memory 404.

As a specific method for this, the following method can be used. On the field area image, pixels included in the field area are circulated, and the color component of each pixel is not included in the field color range, or the value of the hue component H deviates from the field center color (field color distribution Is normalized to a standard deviation) or more, or when the ratio of the variance of the luminance component V of the pixel in the vicinity of the pixel of interest around the field center color to the variance of the field color distribution is a certain value or more. In this case, the pixel of interest is determined to belong to the player area, and the person area image frame memory 40 is determined.
The value of the pixel at the corresponding coordinates in 4 is set to 1. Otherwise, the value is set to 0.

When such processing is executed for all pixels, a binary image showing a person region as shown in FIG. 7 is obtained in the person region image frame memory 404.

The person detecting section 204 includes an image in the image frame memory 401 and a person area image frame memory 404.
Player area image inside, person color model file system 30
2, the color model information of the person in FIG. 2 is input, a person is detected from the image, the position, size, and attribute of the person are acquired and stored in the person information memory 405.

The following method can be used as a specific method for that. First, the person area image frame memory 4
04, the distance conversion is performed on the player region image read from the center, and the result is stored in the person region image frame memory 404.
Write to. The image after the distance conversion is called a distance image. Each pixel value in the range image has a value of the shortest distance from the pixel to the background (that is, a non-human area). For the method, for example, the method described on pages 576 to 577 of "Image Analysis Handbook" (supervised by Mikio Takagi, Hirohisa Shimoda, University of Tokyo Press) can be used.

Next, a set of pixels whose distance value locally has a peak is detected in the distance image. As a method of this, a method of circulating through each pixel, comparing the distance value of the pixel of interest and a neighboring pixel, and detecting the pixel as a peak when the distance value of the pixel of interest is larger than any of them. Available. Subsequently, for each detected peak, a circumscribed rectangle that fits the human area surrounding the peak pixel is calculated. For example, the person area 70 shown in FIG.
With respect to 2, a peak pixel is obtained as shown by a point 703, and a circumscribed rectangle of the person area is obtained as shown by 701.

As one of the methods, as shown in FIG. 8B, a rectangle having a width and a height twice as long as the distance value with the peak pixel at the center is considered, and the upper and lower sides of the rectangle are considered. Are moved upward and downward, respectively, the ratio of the pixels that are returned is included in the human area is calculated, and when the ratio falls below a certain ratio, the movement of the side is stopped. The same operation is performed on the left and right sides. The resulting rectangle is determined as a circumscribed rectangle, and the coordinates of the center position, width,
The height is stored in the personal information memory 405 as personal information.

Further, from the persons stored in the person information memory 405, the area of the overlap of the circumscribed rectangles of the different persons is checked, and if the overlap is remarkable, the person information of the smaller area is stored in the person information memory. 405.
As the method, when the ratio between the area of the overlap and the area of the larger circumscribed rectangle is equal to or smaller than a predetermined value, a method of erasing the smaller area can be used. Also, if there is an overlap, by erasing the circumscribed rectangle with the smaller area,
A method of calculating the ratio of the lost area to the area of the other circumscribed rectangle and erasing the smaller area when it is less than a certain value can be used. Note that a method other than the above method can be used.

Next, as a result of the above processing, the attribute of the person existing in the person information memory 405 is determined.
The attribute information is also added to the person information memory 405. For this purpose, a color model of a person who is a model registered in advance is read from the person color model file system 302, the color model is compared with the detected color of the person, and the attribute of the best matching model is determined as the attribute of the person. .

As one of the realizing methods, the following color information is used. In this example, as shown in FIG. 9, the person region is divided in the vertical direction, and the average value of each color component is calculated for the person region in each divided region. With these color sets,
Similarly, a mean square error between each model whose attribute is designated by the user and the color set of the person is calculated, and the attribute of the model having the smallest error is set as the attribute of the detected person. As a model, in addition to the players and referees of both teams,
By preparing a white line on the field or a bald area of turf, and registering it in the person color model file system 302, if it is determined that the model is closest to the white line or the bald model, the information of the person is deleted. It can be deleted from the person information memory 405, thereby reducing the possibility of erroneously detecting an area such as a white line or baldness as a person.

The transition cost calculation unit 205 calculates the transition cost of a person between frames using the person information detected in the current frame and the previous frame and stored in the person information memory 405 as input. The data is stored in the matrix memory 406.

The transition cost calculation unit 205 performs the processing on the second frame or more in the frame range of the video to be processed. The transition cost is a cost when it is considered that a certain person in the previous frame has moved to a certain person in the current frame, and the number of persons in the previous frame and the number of persons in the current frame are respectively the number of rows and the number of columns. Transition matrix memory 40 as a matrix format
6 is stored.

In this example, two types of costs, a moving cost and a change cost, are calculated as the costs, and the results are stored in the transition cost matrix memory 406, respectively. The moving cost is a cost related to the distance of a person between frames, and the change cost is a cost related to a change in the shape and color distribution of a person between frames. It is considered that those having low costs have actually moved.

The following method can be used as one method of calculating the moving cost. The person i previous of the upper frame, the movement cost D (i, j) of between a current frame j is the coordinate of the person i (x _i, y _i), the coordinates of the person j (x _j, y _j ),
The width of the person i, the height (w _i, h _i), the width of the person j, the height (w _j, h _j) When,

[0060]

(Equation 1)

It can be calculated as follows. Here, V _x and V _y are the shift widths of the images generated between the frames due to the movement of the viewpoint of the camera. In the case of a video taken by a fixed viewpoint camera,
These and 0, in the case of an image taken by means of the view camera processed is the average amount of movement (in pixels) is calculated by a method such as matching between frames of V _x, V _y as.

Since the image is normally taken not from directly above the field but from the side, the distance in the depth direction is expressed on the image as being smaller than the distance in the left and right direction. Therefore, the accuracy of the moving cost can be improved by correcting the moving component in the horizontal direction by the height of the person and correcting the moving component in the vertical direction by normalizing the width of the person.

After the calculation of the moving cost, in order to simplify the subsequent calculation, if the moving cost is a fixed value or more, both the moving cost and the change cost are set to infinity. Also, even when the moving cost is below a certain value, the area ratio of the circumscribed rectangle between the person in the current frame and the person in the previous frame is calculated.
If the area ratio is extremely large or small, the change cost is set to infinity. This has an effect of identifying a dust area other than a person in many cases because the dust area is obtained as a small area.

For each person in the current frame and the previous frame that satisfies these conditions, the change cost is calculated using template matching. This calculates the square error of the color component between images when the center of this template is applied to the person's area of the previous frame using the current frame as a template, and the center position of the template is set in the person's area of the previous frame. While shifting, a position where the square error is minimized is searched, and the smallest square error is set as the change cost. Note that measures other than the squared error can be used.

The inter-frame correspondence determination unit 206 receives the transition cost stored in the transition cost matrix memory 406 as input, determines the correspondence between persons between frames, and uses the determined correspondence between persons as a person state table. The data is stored in the memory 407. Also, the person state table memory 4
07 is output to the person detection result file system 303.

According to the present invention, the correspondence between a person between the current frame and the previous frame and the correspondence between the person between the current frame and the next frame are respectively shown in FIG.
Recognize by dividing into two states. Including the current frame and the preceding and succeeding frames, there are 16 states. As the symbols for identifying the corresponding relationships, the relationship with the previous frame and the relationship with the subsequent frame
Each connection state number is used.

In the correspondence relationship with the previous frame, the connection state number is set to 1 when a person not existing in the previous frame appears, and the connection state number is set to 2 when a simple person moves. Represent. One person area in the previous frame is
In the case where the current frame is split, the split half of the current frame is represented by a connection state number 3. When a plurality of person areas are combined into one in the previous frame, a connection state number 4 is assigned.

The same number is used for the relationship with the next frame. When the person disappears (moves out of the field area) such that the person in the current frame does not exist in the next frame, the connection state is changed. Number 1 is given, 2 is given if the person in the current frame moves simply, 3 is given if the person in the current frame splits, and the person in the current frame merges with another person in the next frame. When one area is formed, connection state number 4 is given.

In the inter-frame correspondence determination section 206, the above-described correspondence between frames is sequentially determined.
The determined person state is stored in the person state table memory 40.
7 is stored in a table format. This is called a “person state table”. The information includes, for each person detected on one image frame as shown in FIG.
Number, connection status number with the previous frame (forward connection status number), connection status number with the next frame (rear connection status number), person number corresponding to the previous frame (forward corresponding person number), person corresponding to the next frame Number (backward corresponding person number).

The ID number is a number unique to a person. If the same person exists over a plurality of frames, the same number is assigned to the corresponding person in those frames. The person number is a serial number given to a person detected on one frame, and specifies a line number in a table in the person state table memory 407. The person number corresponding to the previous frame and the person number corresponding to the next frame are numbers indicating which person in the current frame corresponds to the person in the previous frame and the next frame.

In the present invention, it is assumed that the correspondence shown in FIG. 12 is prohibited. There are (a) Z-type and (b) X-type in this forbidden correspondence, which means that when two people run side by side in two frames, one It stipulates that the exchange of another person (hidden behind) is not performed between two persons who are adjacent in the frame. If a table structure such as the person state table shown in FIG. 11 is used, such a prohibited correspondence cannot be expressed.

For each frame of the video data provided by the video storage file system 301, the correspondence between persons before and after the frame is determined, and the result is stored in the person state table memory 407 as an algorithm.
In the present invention, the following method can be used.

First, for the video data, a user gives a frame number at which processing is started, a frame number at which processing is ended, and a step number indicating an interval at which processing is performed.
In this method, the person state table memory 407 holds a person state table for two frames. Respectively
And the second person state table.

FIGS. 13 and 14 show how the tracking process proceeds and the state of the person is determined for convenience. FIG. 13 (a) shows how the contents of the person status table are described for one person on a certain frame, with the ID number at the top of the circle and the lower left of the circle. The connection state number with the previous frame (forward connection state number) and the lower right part of the circle represent the connection state number with the next frame (rear connection state number). If undecided, each will be blank. The numbers in italics indicate the numbers assigned in the corresponding frame, and the numbers in the Roman type indicate numbers that have already been determined. Further, when the circle is formed by a dotted line, it means that the state and the like have already been determined, and the contents have been output and have been deleted from the person state table. In addition, the corresponding person in the previous frame and the corresponding person in the next frame are shown as the origin of the arrow connected to the left side of the circle and the tip of the arrow extending from the right side of the circle. Here, the correspondence (connection) between persons includes a dominant correspondence (strong connection) and a weak connection, and the strong connection indicates movement of the same person and is indicated by a thick line. The person indicated by this is the person stored in the corresponding column of the person status table. In addition, a person in a weak connection represents a case where another person is united and hidden behind, or a case where a person hidden behind comes out in the next frame.

First, at the stage of the start frame (frame 1), it is assumed that all persons have newly appeared and the first
Is written in the front connection state number column of the person state table of FIG. Also, a new ID number is assigned to the ID number field. In addition, -1 indicating that there is no corresponding destination is written in the front corresponding person number column. This state is shown in FIG.

Next, the processing shifts to the processing of frame 2. At that time, a second person state table is secured in the person state table memory 407. First, for each person in the current frame (frame 2), the person with the smallest total transition cost is selected among the persons in the previous frame (frame 1). Here, the total transition cost is calculated by using the movement cost and the change cost stored in the transition cost matrix memory 406 as follows: total transition cost = movement cost + w × change cost. Here, w is a weighting coefficient. In this case, upper limits E1 and E2 are set for the allowable moving cost and the change cost, and if these conditions are not satisfied, it is assumed that there is no correspondence.

At the stage where this process is completed, FIG.
As shown in (a), there is a case where there is a correspondence from one person in the previous frame to a plurality of persons in the current frame. In this case, the ID of the person in the previous frame is inherited by the person in the current frame, and the ID number is recorded in the column of the person in the second person state table, with the correspondence having the smallest total transition cost as the dominant correspondence. . At the same time, the corresponding person number of the current frame is entered in the backward corresponding person number column of the previous frame, and the corresponding person number of the previous frame is entered in the forward corresponding person number column of the current frame. In addition, for the person of the current frame that is not dominantly connected and is weakly connected, the corresponding person number of the previous frame is entered in the corresponding person number column in the forward direction.

Next, if there is a person in the previous frame who has not been established in correspondence with the person in the current frame, for each of the persons, the person in the current frame may be moved in the transition cost. Cost limit E
If condition 1 is satisfied and the smallest person exists, the person number of the current frame is entered in the backward corresponding person number column of the previous frame.

Next, the person in the current frame has
A new ID number is assigned to a person who has not been assigned a D number, and recorded in the corresponding column of the second person state table. Based on the correspondence of persons between the frames established above, the rear connection state number of the person in the previous frame is determined, entered in the corresponding column of the first person state table, and the front connection state of the person in the current frame is determined. The number is determined and entered in the corresponding column of the second person status table.

Finally, the information on the state of the person in the previous frame (frame 1) recorded in the first person state table is output to the person detection result file system 303, and the first
Erase the contents of the person status table.

Next, the process proceeds to frame 3 and the same processing as in the case of frame 2 is performed. This is shown in FIG. Here, the state of the person in frame 3 is stored in the first person state table. As a result of the process at the stage of frame 3, the status information of the person in frame 2 in the second person status table is output to the person result file system 303. At this time, the output is performed in a form that is added to the information output in the stage of the frame 2.

The above processing is executed for each step of the designated frame until the end frame. In the processing at the end frame stage, -1 is assigned to the backward corresponding person number, and 1 is assigned to the backward connection state number, assuming that all persons disappear, and the content is added to the person result file system 303. Output. The above method is an example, and other methods can be used.

The person transition search unit 207 reads the person detection result in the person detection result file system 303 and
The contents are stored in the person transition structure memory 408, and further, a search for a continuous person over a plurality of frames is performed using the contents as an input, and the result is recorded in the tracking output file system 304.

The person transition search unit 207 aims to output information on the position and size in each frame obtained by tracking each person on an image, which is the final output of the present system.

The following method can be used as an example of the realizing method. First, the person detection result file system 303
From the person detection structure memory 4
08 stores the contents. The data output from the inter-frame correspondence determination unit 206 is stored in the person detection result file system 303. This is data in which a person is detected at certain steps from the start frame to the end frame of the video data in the video storage file system 301 and the correspondence between the frames is described.

Here, the data structure in the person transition structure memory 408 has a graph structure as shown in FIG. FIG.
Reference numeral 5 denotes a frame number on the horizontal axis and an ID number on the vertical axis. Each person is represented as a vertex, and the correspondence between persons between frames is represented as an edge. Using the graph data in the person transition structure memory 408, the person transition search unit 207 performs a search over a plurality of frames of each person and outputs moving trajectory data for each person.

As one of the concrete realizing methods, the following method can be used. First, each vertex is visited in the graph data in the person transition structure memory 408,
A search is made for one whose forward connection state number is 1. This corresponds to newly appearing persons, and a test is performed to determine whether the following conditions are met, starting from each of those persons.

Condition 1: (the backward connection state number is 1; disappears in the next frame) Condition 2: (the backward connection state number is 4; unite in the next frame) Condition 3: (current The ID number is different from the ID number of the corresponding person in the next frame; it is not the dominant correspondence. ・ Condition 4: (Condition 1) or ((Condition 2) and (Condition 3)) ・ Condition 5: Not applicable to (Condition 4) When the condition 5 is satisfied, the process proceeds to the person corresponding to the rear, and a test is performed to determine whether the condition is satisfied. The persons who have passed in the search process are stored as a list. Condition 4
If satisfies, the search starting from the corresponding person is stopped. In this case, a list of persons recording the trajectory of the subsequent persons is generated for the person serving as the starting point, and information on the coordinate values and the size on the image is output to the tracking output file system 304 for each frame. Is done.

Next, if the person is a person who has branched off from another person and is not a dominant person with the corresponding person in the previous frame, the following condition 6 and condition 6: (forward connection state number Is 3; branching in the previous frame) and (the current ID number is different from the ID number of the corresponding person in the previous frame; not the dominant correspondence) are graph data in the person transition structure memory 408. A test is performed on each of the persons who have searched and matched from the inside, and whether or not the above-mentioned conditions 4 and 5 are satisfied, with the respective persons as starting points. This processing is the same as the processing described above. In the above, as a result of searching for the trajectory of the player starting from each person, the length of the list of the trajectories obtained for the persons is checked, and the output of the trajectory whose length is extremely short is stopped, so that the field is output. It is possible to reduce the number of times that an object other than a person such as the upper dust is erroneously detected.

When the trajectory of each person is output, the front connection state number at the start point of the person and the rear connection state number at the end point are also output.

By performing the search processing described above, for example, the movement trajectory of each person as shown in FIG. 16 is obtained from the state transition structure of the person as shown in FIG.
The correspondence between persons can be clearly described.

In the above-described method, the data of the result of the detection of the person and the correspondence between the frames is read via the person detection result file system 303. Data can also be written from the state table memory 407 to the person transition structure memory 408.

[0093] The display unit 1000 reads data relating to the movement trajectory of the person recorded in the tracking output file system 304 and outputs the data to the display device.
As a form of display, for example, as shown in FIG. 16, a time section in which each person exists on the image, a point of time of appearance and disappearance, and a relation such as union and division among a plurality of persons are represented in a graph structure. A mode in which display is performed can be used.

Further, as shown in FIG. 17, a circumscribed rectangle of the detected person is displayed on the image in the video (FIG. 17).
801), the image brightness in the circumscribed rectangle is increased (802 in FIG. 17), and for a plurality of past frames,
The trajectory of the movement of the person can be displayed by superimposing a circumscribed rectangle or an arrow (803 in FIG. 17).

Also, by giving the name and uniform number of each person manually, the person can be displayed with the number and name attached (804).

Furthermore, when the appearance, disappearance, uniting, and separation occur, the color of the circumscribed rectangle is changed and displayed, so that the user can easily recognize the state of the person.

The processing of the processing section 200 shown in FIG. 1 can be realized by, for example, a CPU and a software program (not shown).
It can be installed using various recording media and communication lines. The CPU loads the installed software program into the memory 400 and executes the loaded software program, thereby realizing the above-described series of methods.

[0098]

As described above, according to the present invention, an object is detected in each image frame in a video, the transition cost between objects between image frames is calculated, and a variety of one-to-many, many-to-one, etc. By performing the matching process that introduces the character, and using the results to determine the state of movement, coalescence, division, and disappearance of the object, it is possible to create an image in which multiple people are crowded like a soccer game On the other hand, it enables stable tracking of an object and output of information on the state of merging and separation of a plurality of objects.

Also, even when the shape of an object greatly changes between frames, which is a drawback of the conventional tracking method using template matching, the present invention reduces the possibility of movement for all combinations between objects between frames. Since the examination and the transition cost are calculated, accurate tracking is possible even for an object having a large change in the shape of the object.

[Brief description of the drawings]

FIG. 1 is a diagram showing a system configuration example of the present invention.

FIG. 2 is a flowchart showing a processing example of the system shown in FIG. 1;

FIG. 3 is a diagram illustrating a method of determining a histogram and a field color range.

FIG. 4 is a diagram showing an object on an image frame.

FIG. 5 is a diagram illustrating a field area shaping process.

FIG. 6 is a diagram illustrating an example of an image indicating an extracted field region.

FIG. 7 is a diagram illustrating an example of an image indicating an extracted person region.

FIG. 8 is a diagram illustrating a person region and a circumscribed rectangle thereof.

FIG. 9 is a diagram illustrating acquisition of color information for determining a person attribute.

FIG. 10 is a diagram illustrating a connection relationship between a person and front and rear frames.

FIG. 11 is a diagram showing an example of data stored in a person state table.

FIG. 12 is a diagram showing a forbidden correspondence.

FIG. 13 is a diagram illustrating a state of progress of a tracking process.

FIG. 14 is a diagram showing the progress of a tracking process.

FIG. 15 is a diagram showing a state of a person state transition structure.

FIG. 16 is a diagram illustrating an example of a person transition structure obtained by searching for a person transition.

FIG. 17 is a diagram illustrating an example of display of a tracking result of a person.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 input unit 101 imaging device 102 video image storage / reproduction device 103 image capture device 200 processing unit 201 histogram calculation unit 202 field region extraction unit 203 person region extraction unit 204 person detection unit 205 transition cost calculation unit 206 inter-frame correspondence determination unit 207 person Transition search unit 300 file system 301 video storage file system 302 person color model file system 303 person detection result file system 304 tracking output file system 400 memory 401 image frame memory 402 histogram memory 403 field area image frame memory 404 human area image frame memory 405 Person information memory 406 Transition cost matrix memory 407 Person state table memory 408 Person transition structure memo 1000 display unit

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (Reference) G06T 7/60 150 G06T 7/60 150J H04N 7/18 H04N 7/18 K (72) Inventor Masashi Morimoto Chiyoda-ku, Tokyo 2-3-1 Otemachi Nippon Telegraph and Telephone Corporation (72) Inventor Haruhiko Kojima 2-3-1 Otemachi, Chiyoda-ku, Tokyo F-term within Nippon Telegraph and Telephone Corporation 5B057 BA02 CA01 CA08 CA12 CA16 CB01 CB06 CB08 CB12 CB16 CC03 CD01 CE09 CE11 CE12 CH11 CH12 DA07 DA08 DB02 DB06 DB08 DB09 DC03 DC04 DC23 5C054 FC04 FC08 FC12 FC13 FC16 GB12 GB14 GB17 HA05 5L096 AA02 AA06 CA02 EA14 EA17 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA37 FA15 FA37 FA37 FA37 FA15 FA37 FA37 FA37 FA66 FA69 GA41 HA03 HA05 LA05

Claims (8)

[Claims] An image analysis system that tracks an object over a plurality of image frames constituting a video and outputs information including the position of the image of the object in each image frame. Means for extracting the image area occupied by the object in the frame; means for obtaining the position and size of the object from the image area occupied by the object; Means for calculating a transition cost when an object of a frame moves to an object of a later frame for a set of objects on two frames, and a correspondence between the objects on the two frames based on the transition cost Means for classifying the relationship and judging the state of the object; and determining the correspondence of the object between the two frames to all the neighbors in the video section comprising a plurality of image frames. Means for tracking the movement trajectory of each target object over a plurality of image frames by using information of a result calculated between adjacent frames. 2. A method for extracting an image area occupied by an object in the image frame, comprising: means for calculating a histogram for each color component of pixel values constituting an image; and a color component value indicating a maximum value of the histogram. Means for calculating a color component value corresponding to a valley adjacent to a peak having the maximum value as a peak, thereby calculating a range of a color component in a background region of the object; Means for calculating the ratio of the pixel values in the neighboring region to the range of the color components in the background region, and the magnitude of the variance of the color components in the neighboring region and the magnitude of the variance of the color components in the entire background region Means for calculating the ratio of the variances, and the ratio at which the calculated variance ratio is equal to or less than a predetermined value and the pixel values in the neighboring area are included in the color component range of the background area is equal to or greater than a predetermined value. If
2. An image analysis system according to claim 1, further comprising means for judging that the target pixel is included in the target area, and extracting the target area which is a set of the pixels as a binary image. . 3. A means for acquiring the position and size of an object from an image area occupied by the object, performing distance change on an image holding the area occupied by the object as a binary image, Means for calculating; means for detecting coordinates indicating a local peak of the distance value in the distance image;
Means for calculating a suitable circumscribed rectangle for the object region centered on the coordinates at which the distance value indicates the peak, and outputting the center coordinates of the object as the center position of the circumscribed rectangle; 3. An image analysis system according to claim 1, further comprising means for calculating and outputting a size of the object as a height. 4. As means for calculating the transition cost,
As the transition cost related to the size of the movement of the object, the distance between the objects on the two frames is calculated by correcting the shift of the entire image between the frames due to the movement of the camera's viewpoint in both the horizontal and vertical directions, A means for normalizing the horizontal distance by the width of the object and normalizing the distance in the vertical direction by the height of the object, and a transition cost related to changes in the shape and color of the object 2. The method according to claim 1, further comprising: superimposing a spatial distribution of pixel values of the object on the object on the other frame, and calculating a value that minimizes an error in the distribution of the pixel values. The image analysis system according to claim 3. 5. A means for classifying a correspondence relationship between objects on the two frames and determining a state of the objects,
For each object on the two frames, a means for determining that there is a correspondence between a set of objects whose transition cost is equal to or less than a predetermined value, and a correspondence between the objects on the two frames, Single object in the previous frame corresponds to a single object in the rear frame, multiple objects in the previous frame correspond to a single object in the rear frame, single object in the previous frame An object corresponds to multiple objects in the rear frame, a state exists in the previous frame but no corresponding object exists in the rear frame, and an object exists in the rear frame but corresponds to the previous frame. Classify as non-existent state and simply move the single object in the previous frame to the single object in the subsequent frame. Move multiple objects in the previous frame to the single object in the subsequent frame. The moving state is determined by combining The state in which a single object in the system moves to multiple objects in the subsequent frame is divided into objects. The state in which the corresponding object does not exist in the previous frame but exists in the previous frame is eliminated. Means for judging a state in which there is no corresponding object on the previous frame but exists on the previous frame as appearance of the object. Image analysis system as described. 6. As means for tracing the movement trajectory of each individual object over a plurality of image frames, data on the existence of the object over a plurality of frames and the correspondence relation of the object between the frames are inputted. Means for storing the detected object in each image frame as a graph-structured data structure in which the vertices are the objects detected in each image frame and edges indicate the correspondence between the objects in adjacent frames; Means for searching the graph to obtain the movement trajectory of the object, and searching for the object having the state immediately after the division, until the state immediately before the disappearance or the united state is detected from the starting point. , Searching from the graph until a state immediately before disappearance or a coalesced state is detected from the starting point,
The image analysis system according to any one of claims 1 to 5, further comprising: means for acquiring a movement trajectory of the object. 7. An image analysis method for tracking an object over a plurality of image frames constituting a video and outputting information including the position of the image of the object in each image frame, wherein each image constituting the video is Extracting the image area occupied by the object in the frame; obtaining the position and size of the object from the image area occupied by the object; Calculating the transition cost when the object in the frame moves to the object in the subsequent frame for the set of objects on the two frames, and the correspondence between the objects on the two frames from the transition cost Classifying the relationship and determining the state of the object; and determining the correspondence of the object between the two frames to all adjacent regions in a video section comprising a plurality of image frames. Tracking the movement trajectory of each target object over a plurality of image frames using information of a result calculated between frames. 8. A recording medium recording an image analysis program for tracking an object over a plurality of image frames constituting a video and outputting information including the position of the image of the object in each image frame. A process of extracting an image area occupied by an object in each image frame forming a video; a process of acquiring the position and size of the object from the image area occupied by the object; For the detected object, a process of calculating a transition cost when an object of a previous frame moves to an object of a subsequent frame for a pair of objects on two frames, and two processes from the transition cost. The process of classifying the correspondence between the objects on the frame and determining the state of the object, and the correspondence of the object between the two frames from the plurality of image frames. Using a result of calculation between all adjacent frames in a given video section to track the movement trajectory of each target object over a plurality of image frames. Recording medium for recording an image analysis program, wherein a program for recording the program is recorded.