JP2013045152A - Dynamic body tracker - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently and exactly track a plurality of dynamic bodies even when the dynamic bodies intersect with one another.SOLUTION: Moving images consisting of consecutive original images P(i) in time series are input in an image input part 110, difference with respect to a background image is taken to generate a dynamic body identification image M(i) for distinguishing background from foreground by a dynamic body identification image generation part 130. A tracker storage part 150 stores blob subsumptive graphics circumscribing contours of individual dynamic bodies in time series. A masking processing part 140 utilizes blob subsumptive graphics of dynamic bodies except a noticing dynamic body Tj at time t(i-1) in the tracker storage part 150 to mask a foreground area of the dynamic body identification image M(i). A blob subsumptive graphic extraction part 170 extracts contours of the foreground area except a masked part as candidate blob subsumptive graphics. A tracker registration part 160 selects a subsequent blob subsumptive graphic Bj(i) with respect to a preceding blob subsumptive graphic Bj(i-1) of the noticing dynamic body Tj from among the candidate blob subsumptive graphics and registers the selected blob subsumptive graphic in the tracker registration part 150.

Description

  The present invention relates to a moving object tracking device that detects a plurality of moving objects and tracks them on a screen with respect to a moving image composed of a set of still images continuously given in time series.

  In a surveillance system using a video camera or the like, a technique for automatically detecting and tracking a moving object (moving object) existing in a surveillance area has become important. If automatic detection of a moving object becomes possible, it can be used for processing for identifying whether the object is a vehicle or a human. Furthermore, if the detected moving object can be tracked, the desired moving object can be zoomed in and tracked, which can be used for monitoring a suspicious person or the like.

  The most common method used for moving object detection is to compare the original image in units of frames given in time series with a background image prepared in advance, and obtain the difference between them to change over time. This is a method for recognizing the area that is being used. For example, the following Non-Patent Document 1 discloses a method of comparing an original image and a background image in units of pixels and determining whether or not the pixel is a pixel constituting a moving object. Specifically, if the pixel values of both pixels to be compared are similar, the pixel is a pixel constituting the background, and if not, the pixel is a pixel constituting a moving object. A determination is made. At this time, in the case of a color image, pixel similarity determination is performed by plotting coordinate points corresponding to both pixel values in a three-dimensional color space, and one coordinate point enters a region near the other coordinate point. It is done by checking whether or not.

  Further, in Patent Document 1 below, an input image in units of frames is divided into blocks, an average value of pixel values is obtained for each block, and the block is moved by comparing with an average pixel value of a background image. A method for determining whether or not an object is included is disclosed, and Patent Literature 2 discloses a method for appropriately calculating a difference from a background image and a method for performing template matching even when illumination fluctuation occurs. A method for performing accurate motion detection has been proposed. On the other hand, Patent Document 3 proposes an apparatus that performs moving object tracking by performing imaging using a plurality of cameras.

  Further, Patent Document 4 discloses a method of calculating the color feature amount of the detected moving object and performing tracking of the moving object using this color feature amount. Patent Document 5 divides the original image into sub-partitions. A method of tracking a moving object based on the closeness of the degree of relationship between adjacent sub-parts is disclosed. Patent Document 6 discloses a method of predicting the position of a current moving object from the positions of moving objects in a plurality of past frames and tracking a moving object candidate closest to the predicted position. A technique for tracking a plurality of moving objects without any trouble even when the moving paths of the moving objects intersect is disclosed.

JP-A-7-302328 JP 2011-60167 A JP 2007-142527 A JP 2000-48211 A JP 2007-287006 A JP 2004-46647 A JP 2010-39580 A

Kim K, Chalidabhongse T H, Harwood D, Davis L S. "Background Modeling and Subtraction by Codebook Construction" Proceedings of International Conference on Image Processing, Singapore. IEEE, 2004. 5: 3061-3064

  There are many cases in which a plurality of moving objects are present simultaneously in a field of view such as a surveillance system. In addition, since each of the plurality of moving objects moves in an arbitrary direction, the movement paths may intersect on the screen. Many of the conventionally proposed moving object tracking devices cannot perform accurate tracking when a plurality of moving objects intersect in this way. This is because when the movement paths of a plurality of objects intersect, on the screen, the objects are merged into a group of objects at the time of intersection, and then separated into a plurality of original objects. This is because the object cannot be recognized correctly.

  The above-mentioned Patent Document 7 discloses a technique for appropriately tracking individual moving objects even when moving objects such as people and cars are overlapped in a photographed image of a camera interphone installed at the entrance. . In this technology, when tracking a plurality of moving objects, it is determined whether or not the moving objects cross each other, and if it is determined that they intersect, the depth relationship between the overlapping moving objects is recognized. Thus, a method for performing correct tracking is adopted. However, a complicated process is required for the process of determining the intersection of moving objects and the process of grasping the depth relationship, and it is difficult to always perform an accurate process in various situations.

  Therefore, an object of the present invention is to provide a moving object tracking device capable of tracking individual moving objects efficiently and accurately even when a plurality of moving objects intersect.

(1) According to a first aspect of the present invention, in a moving object tracking device that detects a moving object from a moving image and tracks the moving object,
An image input unit for inputting a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
A background image providing unit that provides a background image constituting the background of the video image;
A moving object identification image generating unit that compares the i-th original image P (i) with the background image and generates an i-th moving object identification image M (i) that distinguishes the background region and the foreground region;
Based on the i-th moving object identification image M (i), a group of pixels constituting the foreground region is recognized as a moving object, and the inclusion figure of the pixel group belongs to the i-th time series. A blob inclusion figure extraction unit to extract as B (i);
Tracker information configured by arranging multiple blob inclusion figures estimated to be extracted based on the same moving object in time series from the first blob inclusion figure to the end blob inclusion figure is stored for each moving object. A tracker storage unit,
The blob inclusion figure B (i) belonging to the i-th time series extracted by the blob inclusion figure extraction unit is the i-th number of the j-th tracker Tj (where j is a natural number indicating the tracker number). As a member belonging to time series, the tracker registration unit to register in the tracker storage unit,
A masking processing unit that performs masking using a specific blob inclusion figure stored in the tracker storage unit for the foreground region of the moving object identification image generated by the moving object identification image generation unit;
Provided,
The masking processing unit uses the jth tracker Tj stored in the tracker storage unit as the target tracker, and the blob inclusion figure B (i-1) belonging to the (i-1) th time series of the non-target tracker. Is used to generate an individual moving body identification image M (i, j) for the tracker of interest Tj by masking the i-th moving body identification image M (i), and this is used as a blob inclusion figure extraction unit. To provide and
In addition to the function of extracting the blob inclusion graphic B (i) based on the moving object identification image M (i), the blob inclusion graphic extraction unit extracts the masked portion of the individual moving object identification image M (i, j). As a candidate blob inclusion figure for the jth tracker that recognizes a group of pixels constituting the excluded foreground area as a moving object and to include the inclusion figure of the pixel group recognized as a moving object in the i-th time series Add the function to extract,
The tracker registration department
(a) When a blob inclusion figure that is not the last blob inclusion figure is registered in the (i-1) th time series of the jth tracker Tj, this blob inclusion figure is used as the preceding blob inclusion figure. To determine whether there is a blob inclusion figure suitable for following in the candidate blob inclusion figure for the jth tracker, and if so, the candidate blob inclusion figure is placed in the i th time series. The process of registering as the subsequent blob inclusion figure to which it belongs, and the subsequent blob inclusion figure registration process for performing processing indicating that the preceding blob inclusion figure has become the end blob inclusion figure when it does not exist,
(b) After the subsequent blob inclusion graphic registration process is performed for each tracker, the blob inclusion graphic B (i) extracted based on the moving object identification image M (i) is registered by the subsequent blob inclusion graphic registration process. New blob inclusion figure registration process that registers the new blob inclusion figure as the first blob inclusion figure in the i-th time series unregistered tracker when there is a new blob inclusion figure that does not correspond to the subsequent blob inclusion figure When,
Is to do.

(2) According to a second aspect of the present invention, in the moving body tracking device according to the first aspect described above,
The background image providing unit has a function of taking in and storing a still image given from the outside, and provides the still image as a background image.

(3) According to a third aspect of the present invention, in the moving body tracking device according to the first aspect described above,
The background image providing unit has a function of obtaining a simple average or a weighted average of the original images input by the image input unit over a predetermined period in the past, and provides the obtained average image as a background image.

(4) According to a fourth aspect of the present invention, in the moving body tracking device according to the first to third aspects described above,
The moving object identification image generation unit compares the color of the pixel at the specific position on the original image P (i) with the color of the pixel at the specific position on the background image, and the degree of approximation of the colors of both pixels is predetermined. When the condition is satisfied, the pixel at the specific position is set as the pixel in the background area, and when the predetermined condition is not satisfied, the pixel at the specific position is set as the pixel in the foreground area. Is generated.

(5) According to a fifth aspect of the present invention, in the moving body tracking device according to the first to fourth aspects described above,
The blob inclusion figure extraction unit does not recognize a pixel group whose area does not satisfy a predetermined standard as a moving object and does not extract a blob inclusion figure.

(6) According to a sixth aspect of the present invention, in the moving object tracking device according to the first to fifth aspects described above,
The blob inclusion figure extraction unit uses the circumscribed rectangle of the pixel group as the inclusion figure.

(7) According to a seventh aspect of the present invention, in the moving object tracking device according to the sixth aspect described above,
The blob inclusion figure extraction unit pays attention to the individual pixels constituting the moving object identification image M (i) in order,
(a) When the target pixel is a foreground pixel and there is no registration rectangle in the predetermined vicinity range, a process of registering a rectangle that forms the outline of the target pixel is performed.
(b) If the pixel of interest is a foreground pixel and there is a rectangle registered in the predetermined neighborhood range, the registration rectangle is expanded to a circumscribed rectangle including the pixel of interest and registered.
The blob inclusion figure is extracted based on the finally registered rectangle.

(8) According to an eighth aspect of the present invention, in the moving object tracking device according to the sixth or seventh aspect described above,
The tracker storage unit stores the coordinate values of the two vertices located at the diagonal of the circumscribed rectangle as tracker information.

(9) According to a ninth aspect of the present invention, in the moving body tracking device according to the first to eighth aspects described above,
When the masking processing unit generates the individual moving object identification image M (i, j), the blob inclusion figure registered in the (i-1) -th time series of the non-focused tracker is the first of the focused tracker. (I-1) Masking is performed using only the blob inclusion figure located within a predetermined neighborhood range of the blob inclusion figure registered in the first time series.

(10) According to a tenth aspect of the present invention, in the moving object tracking device according to the first to ninth aspects described above,
The blob inclusion belonging to the i-th time series based on the transition of the position of the blob inclusion figure belonging to the (i−1) -th consecutive time series stored in each tracker. When generating the individual moving object identification image M (i, j), the function includes a function for obtaining a predicted position of the figure, and the blob inclusion figure B (i belonging to the (i-1) th time series of the tracker not of interest -1) is subjected to position correction to the predicted position, and masking is performed on the i-th moving object identification image M (i) using the inclusion figure after position correction.

(11) According to an eleventh aspect of the present invention, in the moving body tracking device according to the tenth aspect described above,
The masking processing unit performs the (i−1) th from the reference point at the position of the blob inclusion figure B (i−L) belonging to the (i−L) th (L is an integer of 2 or more) time series. The tip of the vector obtained by extending the length to the reference point at the position of the blob inclusion figure B (i-1) belonging to the time series is multiplied by (L / (L-1)) times. The reference point of the predicted position is used.

(12) According to a twelfth aspect of the present invention, in the moving body tracking device according to the tenth aspect described above,
The masking processing unit belongs to the (i-1) th time series from the blob inclusion figure B (iL) belonging to the (i-L) th (L is an integer of 3 or more) time series. A quadratic curve that approximately passes through the reference points at each position of the total L blob inclusion figures up to the blob inclusion figure B (i-1) is obtained, and a point on the quadratic curve is used as a reference point of the predicted position It is what I did.

(13) According to a thirteenth aspect of the present invention, in the moving body tracking device according to the first to twelfth aspects described above,
When the masking processing unit generates the individual moving object identification image M (i, j), the blob inclusion figure B (i-1) belonging to the (i-1) th time series of the tracker not of interest Then, the contour line is corrected to extend outward by a predetermined amount, and the i-th moving object identification image M (i) is masked using the inclusion figure after the expansion correction.

(14) According to a fourteenth aspect of the present invention, in the moving body tracking device according to the first to thirteenth aspects described above,
When the tracker registration unit performs the subsequent blob inclusion graphic registration process, the jth track inclusion graphic is registered with respect to the preceding blob inclusion graphic registered in the (i-1) th time series of the jth tracker Tj. It is suitable for evaluating the suitability of individual candidate blob inclusion figures extracted for tracker use, and following the preceding blob inclusion figure with the candidate blob inclusion figure with the highest evaluation and the highest evaluation. It is determined that it is a blob inclusion figure and is registered as a subsequent blob inclusion figure.

(15) According to a fifteenth aspect of the present invention, in the moving body tracking device according to the fourteenth aspect described above,
The tracker registration unit compares the preceding blob inclusion figure to be determined and the candidate blob inclusion figure with respect to any one or a combination of position, area, and shape, and performs suitability evaluation based on the similarity. It is what I did.

(16) According to a sixteenth aspect of the present invention, in the moving object tracking device according to the first to fifteenth aspects described above,
When the tracker registration unit registers the subsequent blob inclusion graphic for the jth tracker Tj in the subsequent blob inclusion graphic registration process, the size of the subsequent blob inclusion graphic is registered in advance as a predetermined value. The size of the subsequent blob inclusion figure is corrected so as to be the average size of the number of blob inclusion figures.

(17) According to a seventeenth aspect of the present invention, in the moving body tracking device according to the first to sixteenth aspects described above,
A blob inclusion figure suitable for the tracker registration unit to follow the preceding blob inclusion figure registered in the (i-1) th time series of the jth tracker Tj in the subsequent blob inclusion figure registration process When the candidate blob inclusion figure for the jth tracker does not exist in the candidate blob inclusion figure, the preceding blob inclusion figure is terminated by performing the process of making the i-th time series of the jth tracker Tj unregistered. It shows that it became a blob inclusion figure.

(18) According to an eighteenth aspect of the present invention, in the moving object tracking device according to the first to seventeenth aspects described above,
When the tracker registration unit performs the new blob inclusion figure registration process, the blob inclusion figure B (i) extracted based on the moving object identification image M (i) and the subsequent blob inclusion figure registration process are the i-th time. The subsequent blob inclusion graphic registered in the series is compared, and if both match or one of them includes the other, the blob inclusion graphic B ( The blob inclusion figure whose correspondence cannot be recognized in i) is changed to a new blob inclusion figure.

(19) According to a nineteenth aspect of the present invention, in the moving object tracking device for detecting a moving object for a moving image and tracking the moving object,
An image input unit for inputting a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
A background image providing unit that provides a background image constituting the background of the video image;
A moving object identification image generating unit that compares the i-th original image P (i) with the background image and generates an i-th moving object identification image M (i) that distinguishes the background region and the foreground region;
Tracker information consisting of blob inclusion figures consisting of inclusion figures of pixel groups estimated to be the same moving object in time series from the beginning blob inclusion figure to the end blob inclusion figure, for each moving object A tracker storage to store;
A moving object identification image M (i) is stored using a blob inclusion figure indicating a moving object other than the moving object of interest included in the (i-1) th original image P (i-1) stored in the tracker storage unit. Masking processing unit for performing masking processing on the foreground area of
For the moving object identification image M (i) before and after the masking process, a block that recognizes a group of pixels constituting the foreground area excluding the masked portion as a moving object and extracts the inclusion figure of the pixel group as a blob inclusion figure An inclusion figure extraction unit;
The i-th original image P (i) with respect to the preceding blob inclusion figure indicating the moving object of interest contained in the (i-1) -th original image P (i-1) stored in the tracker storage unit. ) Included in () is extracted from the moving object identification image M (i) masked using the blob inclusion figure indicating the moving object other than the moving object. The candidate blob inclusion graphic is selected from the candidate blob inclusion graphic and registered in the tracker storage unit as the i th time-series subsequent blob inclusion graphic following the preceding blob inclusion graphic. Among the blob inclusion figures extracted from i), a new blob inclusion figure for which the corresponding subsequent blob inclusion figure is not registered is used as a tracker whose i-th time series is unregistered. In, and a tracker registration unit to be registered as a top blob inclusion figure,
Is provided.

(20) According to a twentieth aspect of the present invention, in the moving object tracking device according to the nineteenth aspect described above,
The tracker registration department
Comparing the preceding blob inclusion graphic and each candidate blob inclusion graphic with respect to any one or more combinations of position, area, and shape, and selecting the subsequent blob inclusion graphic based on the similarity,
Of the individual blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the blob inclusion figure registered as the subsequent blob inclusion figure of the i-th time series matches or is included. The blob inclusion figure which is not used is made into a new blob inclusion figure.

  (21) In a twenty-first aspect of the present invention, the moving object tracking device according to the first to twentieth aspects described above is configured by incorporating a dedicated program into a computer.

(22) In a twenty-second aspect of the present invention, in addition to the moving body tracking device according to the first to twentieth aspects described above,
A video camera that captures a video image and gives it to the image input unit;
The blob inclusion figure B (i) successively registered in one selected tracker is sequentially extracted from the tracker storage unit, and the inside of the blob inclusion figure B (i) from the original image P (i) input by the image input unit A partial video presentation unit that presents a partial video image tracking a specific moving object by cutting out and displaying the image of
The monitoring system is configured by providing

(23) According to a twenty-third aspect of the present invention, in the monitoring system according to the twenty-second aspect described above,
The partial moving picture presentation unit expands the blob inclusion figure B (i) outward by a predetermined amount, and cuts out and displays an image inside the expanded inclusion figure.

(24) According to a twenty-fourth aspect of the present invention, in the moving object tracking method for detecting a moving object in a moving image and tracking the moving object,
An image input stage in which a computer inputs a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
The computer compares the background image constituting the background of the moving image and the i-th original image P (i) to distinguish the background region from the foreground region, and the i-th moving object identification image M (i). A moving object identification image generation stage for generating
In the i-th original image P (i), the computer compares the preceding blob inclusion figure including the outline of the moving object of interest included in the (i-1) -th original image P (i-1). A new blob inclusion that recognizes the subsequent blob inclusion figure that includes the outline of the moving object that is estimated to be the same as the moving object of interest and that includes the outline of the moving object that newly appears in the i-th original image P (i). Subsequent / newcomer blob inclusion figure recognition stage to recognize the figure,
And
In the subsequent / newcomer blob inclusion figure recognition stage,
A foreground area of the moving object identification image M (i) is obtained using a blob inclusion figure that includes the outline of a moving object other than the moving object of interest included in the (i-1) th original image P (i-1). Masking processing step for performing masking processing,
In the moving object identification image M (i) before and after the masking process, a group of pixels constituting the foreground area excluding the masked portion is recognized as a moving object, and a figure including the outline of the pixel group is defined as a blob inclusion figure. Blob inclusion figure extraction step to extract;
From the candidate blob inclusion graphic extracted from the moving object identification image M (i) masked using the blob inclusion graphic that includes a moving object other than the target moving object, to the preceding blob inclusion graphic that includes the contour of the target moving object A subsequent blob inclusion shape selection step for selecting a blob inclusion shape suitable for subsequent as the subsequent blob inclusion shape;
Among the blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the new blob inclusion figure in which the new blob inclusion figure in which the corresponding subsequent blob inclusion figure is not recognized is included in the outline of the new moving object is included. A new blob inclusion figure recognition step that recognizes as
Is to do.

(25) According to a twenty-fifth aspect of the present invention, in the moving body tracking method according to the twenty-fourth aspect described above,
In the subsequent blob inclusion graphic selection step, the previous blob inclusion graphic and each candidate blob inclusion graphic are compared with respect to one or more combinations of position, area, and shape, and the subsequent blob inclusion graphic is determined based on the similarity. Make a selection
In the new blob inclusion figure recognition step, among the individual blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the blob inclusion figure recognized as the subsequent blob inclusion figure is matched or included in the relation The blob inclusion figure which is not made into a new blob inclusion figure.

  In the moving object tracking device according to the present invention, a method of masking a region where a moving object other than the target moving object exists when detecting a moving object that inherits a certain target moving object on the time axis among a plurality of moving objects. Therefore, even when a plurality of moving objects intersect, it becomes possible to track each moving object efficiently and accurately.

It is a top view which shows the example of a detection of the moving body about a moving image. It is a figure which shows the basic principle of a general moving body detection. It is a figure which shows an example of the production method of the average image used as an image which shows a background. It is a figure which shows the pixel value transition process of 1 pixel on the average image produced based on the method shown in FIG. It is a figure which shows the pixel value determination process of the general moving body identification image M (i). It is a top view which shows the example which extracted blob inclusion figure B30, B40 based on the moving body identification image M30 and M40 shown in FIG. It is a top view which shows the state which cut out the partial images f30 and f40 using the blob inclusion figure B30 and B40 shown in FIG. It is a top view which shows the original image P10 in which several moving bodies exist (a background is abbreviate | omitted). It is a top view which shows the moving body identification image M10 produced from the original image P10 shown in FIG. It is a top view which shows original image P10, P20, P40 in the time t10, t20, t40 on a time-axis. It is a top view which shows the moving body identification image M10, M20, M40 produced from the original images P10, P20, and P40 shown in FIG. 10, and the blob inclusion figure extracted from these. It is a figure which shows an example of the tracker produced based on the moving body identification image M10, M20, and M40 shown in FIG. It is a top view which shows the actual data of the blob inclusion figure B. It is a top view which shows original image P20, P30, P40 in the time t20, t30, t40 on a time series. It is a top view which shows the moving body identification image M20, M30, M40 produced from the original images P20, P30, and P40 shown in FIG. 14, and the blob inclusion figure extracted from these. It is a figure which shows an example of the tracker produced based on the moving body identification image M20, M30, and M40 shown in FIG. It is a figure which shows another example of the tracker produced based on the moving body identification image M20, M30, and M40 shown in FIG. It is a top view which shows the original images P29 and P30 in the time t29 and t30 on a time series. It is a top view which shows the moving body identification image M29 and M30 produced from the original images P29 and P30 shown in FIG. 18, and the blob inclusion figure extracted from these. A plan view (Fig. (A)) showing an individual moving object identification image M (30, 1) obtained by performing masking processing with a blob inclusion graphic other than the blob inclusion graphic B29a shown in Fig. 19 (a), and extracted from this. It is the top view (figure (b)) which shows the candidate blob inclusion figure B30a. A plan view (FIG. (A)) showing an individual moving object identification image M (30, 2) obtained by performing masking processing with a blob inclusion figure other than the blob inclusion figure B29b shown in FIG. It is a top view (figure (b)) which shows candidate blob inclusion figure B30b. A plan view (Fig. (A)) showing an individual moving object identification image M (30, 3) obtained by performing masking processing with a blob inclusion graphic other than the blob inclusion graphic B29c shown in Fig. 19 (a), and extracted from this. It is a top view (figure (b)) which shows candidate blob inclusion figure B30c. A plan view (Fig. (A)) showing an individual moving object identification image M (30, 4) obtained by performing masking processing with a blob inclusion graphic other than the blob inclusion graphic B29d shown in Fig. 19 (a), and extracted from this. It is the top view (figure (b)) which shows the candidate blob inclusion figure B30d. It is a figure which shows the state which registered the subsequent blob inclusion figure to each tracker based on the candidate blob inclusion figure shown in FIGS. It is a top view which shows the state which accumulated the subsequent blob inclusion figure recognized about each tracker T1-T4 on the moving body identification image M30. FIG. 21 is a plan view (FIG. (A)) showing a state where complete masking has not been performed in the masking process shown in FIG. 20 and a plan view (FIG. (B)) showing candidate blob inclusion figures extracted from this state. It is a top view which shows the original images P39 and P40 in the time t39 and t40 on a time series. It is a top view which shows the moving body identification image M39 and M40 produced from the original images P39 and P40 shown in FIG. 27, and the blob inclusion figure extracted from these. It is a figure which shows the tracker produced based on the blob inclusion figure shown in FIG. 25 is a table showing a correspondence relationship between the subsequent blob inclusion graphic registered in the column of time t30 in the table of FIG. 24 and the blob inclusion graphic on the moving object identification image M30 shown in FIG. 19B. It is a block diagram which shows the structure of the moving body tracking device which concerns on fundamental embodiment of this invention. It is a flowchart which shows an example of the procedure of the blob inclusion figure extraction process by the blob inclusion figure extraction part 170 in the moving body tracking device shown in FIG. It is a top view which shows an example of the image used as the process target of the blob inclusion figure extraction process shown in FIG. It is a top view which shows each step of the process which performed the procedure of the blob inclusion figure extraction process shown in FIG. 32 with respect to the image shown in FIG. It is a top view which shows the method of the suitability evaluation of the following blob inclusion figure by the tracker registration part 160 in the moving body tracking device shown in FIG. It is a top view which shows an example of the method of calculating | requiring the predicted position with respect to the blob inclusion figure which functions as a mask by the masking process part 140 in the moving body tracking device shown in FIG. It is a top view which shows another example of the method of calculating | requiring the predicted position with respect to the blob inclusion figure which functions as a mask by the masking process part 140 in the moving body tracking device shown in FIG. It is a top view which shows an example of the expansion correction with respect to the blob inclusion figure by the masking process part 140 in the moving body tracking device shown in FIG. It is a top view which shows an example of the magnitude correction with respect to the blob inclusion figure by the tracker registration part 160 in the moving body tracking device shown in FIG. It is a flowchart which shows the procedure of the moving body tracking method based on the basic concept of this invention. It is a block diagram which shows an example of the monitoring system comprised using the moving body tracking device based on this invention.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic principle of general motion detection >>
First, the basic principle of general moving object detection will be described. This basic principle is for detecting a moving object (moving object) in a moving image, and here, as a set of pixels having pixel values of three primary colors, a color original image in units of frames continuously in time series. The case where is given will be described as an example.

  Assume that original images P10, P20, P30, and P40 in units of frames are given in this order as shown on the left side of FIG. The illustrated example shows a photographed image by a fixed-point video camera installed on the street, and shows a state in which one vehicle (moving body) passes from right to left. Note that in the case of a general video camera, captured images are output continuously at a cycle of about 30 frames per second, so in practice, a large number of frame images can be obtained at finer time intervals. However, here, for convenience of explanation, consider a case where four original images P10, P20, P30, and P40 are given in order as shown in the figure.

  Looking at these four original images, the original image P10 is only the background of the city, but in the original image P20, a part of the vehicle has entered on the right side, so that the part of the vehicle overlaps the background. ing. Similarly, the original images P30 and P40 are images in which vehicles overlap on the background. Since the background is stationary, it is common to the original images P10 to P40. However, since the vehicle is a moving object, the original images P10 to P40 are different. Here, on each image, the area constituting the background is referred to as the background area K, and the area constituting the vehicle (moving object) is referred to as the foreground area F.

  The purpose of the moving object detection performed here is to distinguish the background area K and the foreground area F from each of the given original images P10 to P40. The images M10 to M40 shown on the right side of FIG. 1 are images showing the distinction between the background region K and the foreground region F with respect to the original images P10 to P40 shown on the left side. Here, the moving object identification images M10 to M40 are shown. I will call it. In the moving object identification images M10 to M40, the hatched area in the figure is the foreground area F (F20, F30, F40), and the white area is the background area K (K10 to K40). Since these moving body identification images M10 to M40 only need to be able to identify two regions, they are given as binary images. For example, if the pixel value “1” is given to the pixels belonging to the foreground area F and the pixel value “0” is given to the pixels belonging to the background area K, the moving object identification images M10 to M40 are 1 bit. It is possible to prepare as image data composed of an aggregate of pixels having the pixel value.

  If a method described later is adopted, a corresponding moving body identification image M can be obtained as a binary image for each of the given original images P in units of frames. After all, the essence of the moving object detection process described here is a binary image (moving object identification image M) indicating the distinction between the background region K and the foreground region F for each original image P continuously given in time series. It can be said that the process of creating.

  In the example shown in FIG. 1, an example in which moving object identification images M10 to M40 are created for each of the original images P10 to P40 is shown. By using these moving object identification images, it is possible to recognize which area on the original image is a moving object, so you can grasp the shape and size of the moving object, or zoom in on the moving object part and check the vehicle number. Or a process for specifying a person can be performed. In addition, it is possible to track a moving object on the screen while zooming up, or to grasp a moving route.

  Of course, it is difficult to create such a moving body identification image from only one original image. For example, when only the illustrated original image P20 is given as a still image, the human brain can recognize the presence of the vehicle from various information, but the presence of the vehicle is detected by the computer from only this one still image. In order to recognize this, processing based on a complicated algorithm is required. Therefore, as described above, a general moving object detection algorithm employs a method of determining a difference area between the original image and the background image and determining the difference area as a moving object area (foreground area F).

  For example, if the original image P10 is designated as a background image in advance, the difference area (different area) is recognized as the foreground area F by comparing the two when the original image P20 is input, as shown in the figure. It is possible to create a moving object identification image M20 (an image that distinguishes the background region K20 and the foreground region F20). In order to recognize the difference area, the original image and the background image are compared in pixel units, and when the pixel values of both pixels are similar, the pixel is assumed to be a pixel constituting the background area K. If not, the simplest method is that the pixel is a pixel constituting the foreground region F.

  As a background image, it is possible to use a still image that has been captured in a state where there is no moving object in advance (for example, the state shown in the original image P10). Are changing from moment to moment, and it is inappropriate to continue using a still image taken at a specific time point as a background image. Therefore, usually, a method is employed in which an average image having an average characteristic of these original images is created based on a plurality of original images input in the past, and this average image is used as a background image. Since the average image is sequentially updated each time a new image is input, even if the lighting environment changes due to factors such as weather and time, the average image functions as an appropriate background image following the change.

  FIG. 2 is a diagram showing a basic principle of general moving object detection. Here, the original images P (1), P (2),..., P (i-2), P (i-1) shown in the upper part of the figure are frame-units continuously given in time series. This is an original image, and so far (i-1) frames of images have been given from the first original image P (1) to the (i-1) th original image P (i-1). The state is shown. On the other hand, the image P (i) shown on the left in the middle of the figure is the i-th original image newly given, and the image A (i-1) shown on the right in the middle of the figure is the past shown in the upper part of the figure Is an average image having average characteristics of the original images P (1) to P (i-1) for (i-1) frames.

  The moving body identification image M (i) for the newly input original image P (i) is created by comparing the original image P (i) with the average image A (i-1). The lower part of FIG. 2 shows the moving object identification image M (i) created in this way. Specifically, the pixel value of each pixel on the original image P (i) is compared with the pixel value of the corresponding pixel on the average image A (i−1). A pixel value (for example, “0”) indicating the background area K (i) is given, and a pixel value (for example, “1”) showing the foreground area F (i) is given to a pixel determined to be outside the similar range. Thus, a moving object identification image M (i) composed of a binary image may be created.

  Subsequently, when the (i + 1) -th original image P (i + 1) is input, the average image A (i) having an average feature up to the i-th original image P (i) input in the past. Is newly created, and the moving object identification image M (i + 1) is created by comparing the original image P (i + 1) and the average image A (i). In this way, since the average image is sequentially updated, even if the lighting environment changes due to factors such as weather and time, it can function as a background image suitable for each time point.

  Note that when creating an average image, it is preferable to perform weighting along a time series, and obtain a weighted average with the latest original image information weighted. For example, if an average is obtained by giving a weight of 1 for an original image input in the past 10 minutes and a weight of 0 for an original image input before that, an average image for the original image obtained in the latest 10 minutes is always obtained. It can be obtained and can cope with a change in the lighting environment. Of course, the weights can be set finely so that the latest original image always has a higher weight.

  After all, in the example shown in FIG. 2, in order to obtain the average image A (i−1), the (i−1) th original image P (i−1) from the first original image P (1) is obtained. ) (I = 1, 2,...), A weighted average value of pixel values for each color of pixels at corresponding positions is calculated, and an image composed of a collection of pixels having the average value is created. That's fine. When the i-th original image P (i) is input, the original image P (i) and the average image A (i-1) are generated in order to create the i-th moving object identification image M (i). Can be compared.

  However, if a moving image is input at a general frame rate of 30 frames per second, even if it is an original image for the past 10 minutes, in order to store all the image data, the image data buffer Considerable storage capacity is required. Therefore, practically, when the i-th original image P (i) is given, the (i−1) -th average image A (i−1) and the i-th original image P ( Based on the two images i), a method of creating the i-th average image A (i) is adopted. FIG. 3 is a diagram illustrating an example of an average image creation method based on such a method.

First, when the first original image P (1) is given, the original image P (1) is processed as it is as the first average image A (1). Then, every time the i-th original image P (i) shown in the upper right of FIG. 3 is inputted (i = 2, 3,...), The (i−) shown in the upper left of FIG. 1) The i-th average image A (i) shown in the middle part of FIG. 3 is shown in the lower part of FIG.
a (i) = (1-w) .a (i-1) + w.p (i) Formula (1)
It may be created using the following arithmetic expression. Here, a (i) is a pixel value of a predetermined color of a pixel at a predetermined position of the average image A (i), and a (i-1) is a pixel value of the pixel at the predetermined position of the average image A (i-1). The pixel value of the predetermined color, p (i) is the pixel value of the predetermined color of the pixel at the predetermined position of the original image P (i), and w is a parameter (w <1) indicating a predetermined weight.

  In short, the pixel value a (i−1) of the (i−1) -th average image A (i−1) calculated for the original image up to the original image P (i−1) given immediately before, For the pixel value p (i) of the i-th original image P (i) that is newly given, an average value considering the weight w is calculated, and the calculated value is used as the i-th average image A (i). The process of setting the pixel value a (i) is performed. Thus, every time a new original image P (i) is input, the average image A (i) is updated, and only the immediately preceding average image A (i-1) is used for the update. By doing so, it is not necessary to store and hold past original image data for several frames, so that it is possible to greatly save the capacity of the buffer necessary for processing.

  As shown in FIG. 3, as the value of the weight w is increased, the pixel value of the latest original image P (i) greatly affects the pixel value of the average image A (i). FIG. 4 is a diagram showing a pixel value transition process of one pixel on the average image created based on the method shown in FIG. In this example, a specific example in which the weight w = 0.01 is shown.

  For example, if the pixel value of the pixel at the specific position of the first (i = 1) original image P (1) is p (1) = 100, the specific of the first average image A (1) is specified. The pixel value a (1) of the pixel at the position is a (1) = 100 as it is. Subsequently, assuming that the pixel value of the pixel at the specific position of the second original image P (2) is p (2) = 100, the pixel at the specific position of the second average image A (2). The pixel value a (2) is calculated as a weighted average “0.99 × 100 + 0.01 × 100” between a (1) and p (2), so that a (2) = 100.

  In the illustrated example, since the pixel value at the specific position of the original image remains 100 from p (1) to p (4), the pixel value at the specific position of the average image is also a (1) to a Maintain 100 until (4). However, in the fifth original image P (5), the pixel value a (5) of the fifth average image A (5) is changed in the lower part of FIG. As shown in the equation, a (5) = 101 is obtained by calculating the weighted average “0.99 × 100 + 0.01 × 200” of a (4) and p (5). Further, since the pixel value p (6) = p (7) = 200 is maintained in the sixth original image P (6) and the seventh original image P (7), the lower part of FIG. As shown in the equation, in the sixth average image A (6) and the seventh average image A (7), the pixel value a (6) = 102 and the pixel value a (7) = 103 gradually increase. doing.

  As in this example, when the weight w is set to a value of about 0.01, even if the pixel value of the original image changes abruptly, the pixel value of the average image does not change immediately, and the pixel value of the original image It will approach you slowly. Therefore, as in the example shown in FIG. 1, even if a moving image through which the vehicle passes is given, the transient pixel value change does not cause a large change in the pixel value of the average image, and the average image It can serve as an image. On the other hand, if the lighting environment changes, such as when the time changes from daytime to dusk, the change persists across many frame images, so the average image pixel value also changes following the change. become. Therefore, an average image suitable for a daytime background image is obtained during the daytime, and an average image suitable for a sunset background image is obtained at dusk.

  The moving body identification image M showing the moving body region can be created by comparing individual original images with the average image created by such a method as a background image. FIG. 5 is a diagram showing a pixel value determination process of the moving object identification image M created in this way. The moving object identification image M (i) for the i-th original image P (i) is a binary image composed of a collection of pixels having pixel values determined by the following method. That is, the pixel value m (i) of the pixel at the specific position of the moving object identification image M (i) is equal to the pixel value p (i) of the pixel at the specific position of the original image P (i) and the average image A (i− 1) is determined by comparison with the pixel value a (i−1) of the pixel at the specific position, and if both pixel values are within the similar range, the pixel value indicating that the pixel is in the background region K (for example, , “0”), and if both pixel values are outside the similar range, a pixel value (for example, “1”) indicating a pixel in the foreground area F is given.

  If the original image is a color image having pixel values of three primary colors (for example, in the case of the RGB color system, pixel values of three colors R, G, and B), the original image P (i) is formed. The pixel value p (i) of each pixel to be processed and the pixel value a (i) of each pixel constituting the average image A (i) are each composed of three independent values for each color. . Then, the calculation of Expression (1) is executed independently for each color. For example, the pixel value a (i) of a pixel at a specific position on the average image A (i) is composed of three sets of pixel values, an R color pixel value, a G color pixel value, and a B color pixel value. The R pixel value is a value obtained as a weighted average of the R pixel values of the past original image.

  Therefore, the comparison between the pixel value p (i) and the pixel value a (i−1) in FIG. 5 is not a simple comparison between two values, but a comparison between three sets of pixel values and three sets of pixel values. become. Naturally, the pixel similarity determination is a process for determining whether or not colors composed of combinations of these three pixel values are similar. Such similarity determination of two colors can be performed based on the magnitude of the Euclidean distance in the color space, for example. That is, when the pixel values R, G, and B are plotted on the coordinate space with the two colors to be compared as points, and the Euclidean distance between the two points is equal to or less than a predetermined reference value, If it is similar or exceeds the reference value, it may be dissimilar.

<<< §2. Moving object tracking method using blob inclusion figure >>
In §1, the basic principle of general moving object detection is described. Here, a method for tracking the moving object detected in this way will be described. Tracking of a moving object is to follow a moving object detected in units of individual frames in time series. For example, in the example shown in FIG. 1, when the moving object identification images M20, M30, and M40 are individually viewed, the foreground areas F20, F30, and F40 can be recognized as moving objects, respectively. F20, F30, and F40 are grasped as time series information of the same moving object. In general, a group of pixels constituting each foreground region F20, F30, F40 corresponding to a moving object is called a blob, and moving object tracking is a time series of blobs indicating the same moving object. It is to grasp side by side.

  A blob included in an original image of a certain frame can be defined by its outline. For example, the foreground area F30 included in the moving object identification image M30 in FIG. 1 is a blob indicating the area of the moving object (vehicle) in the original image P30, and if there is information on the outline of this blob, the original image P30 It is possible to cut out only the moving body (vehicle) part from Accordingly, the result of moving object tracking can be shown by arranging information on the outline of the blob in time series.

  However, in reality, the outline of the blob is a very complicated figure, and a considerable amount of storage capacity is required to store such outline information in units of individual frames. In general applications, such complicated contour information is rarely required. Therefore, in practice, a method of representing the position, shape, and size of the moving object by the circumscribed figure of the blob is adopted. In the present application, such circumscribed figure is referred to as “blob inclusion figure”. As the circumscribed figure, any figure may be adopted, but usually a quadrangle is used. Therefore, also in the basic embodiment described below, a circumscribed rectangle of a blob is defined as a “blob inclusion graphic”, and the position, shape, and size of a moving object are represented by this blob inclusion graphic.

  6 (a) and 6 (b) are plan views showing examples in which blob inclusion figures B30 and B40 (rectangles indicated by bold lines) are extracted based on the moving body identification images M30 and M40 shown in FIG. A blob inclusion figure B30 shown in FIG. 6 (a) is a circumscribed rectangle of the foreground area F30 (blob) included in the moving object identification image M30, and a blob inclusion figure B40 shown in FIG. 6 (b) is added to the moving object identification image M40. It is a circumscribed rectangle of the foreground area F40 (blob) included. The blob inclusion figures B30 and B40 made of such circumscribed rectangles are simple geometric figures, and can be expressed with extremely small amounts of information, and roughly indicate the position, shape, and size of the moving object. Can do. Of course, the information of the blob inclusion figure is less accurate than the information of the outline of the blob, and only a rough aspect ratio can be shown, especially regarding the shape. However, in practice, it is sufficient if the moving object is tracked by the blob inclusion figure having such rough information.

  For example, looking at the transition of the blob inclusion figure from FIG. 6 (a) to FIG. 6 (b), it can be recognized that the horizontally long blob inclusion figure is moving from the right to the left in the lower part of the screen. If this is compared with the transition from the original image P30 to P40 in FIG. 1, it is possible to track the vehicle as a moving object. 7A and 7B show a state in which partial images f30 and f40 are cut out from the original images P30 and P40 using the blob inclusion figures B30 and B40 shown in FIGS. 6A and 6B, respectively. FIG. The figure shown in the lower part of the figure shows a state in which the cut out partial images f30 and f40 are enlarged and displayed on the full screen of the display.

  The aspect ratio (aspect ratio) of the display screen is normally fixed, whereas the aspect ratio of the rectangle constituting the blob inclusion figure is arbitrary depending on the shape of the moving object, so that the blob inclusion figures B30 and B40 However, the moving object (vehicle) is always zoomed by cutting out so as to display the blob inclusion figures B30 and B40 almost completely on the screen. It will be possible to present videos that are tracked in the up state. Therefore, in the case of an application where a detected moving object is tracked and displayed in a zoomed-up state in a monitoring system or the like, if information that arranges blob inclusion figures consisting of circumscribed rectangles of the moving object in time series is obtained, the information Is sufficiently useful as information indicating the results of moving object tracking.

  The object of the moving body tracking apparatus according to the basic embodiment described here is to obtain information (such information is referred to as a tracker as will be described later) in which such blob inclusion figures are arranged on the time axis. However, the example shown in FIG. 1 is an example of a simple moving image in which a single moving object (vehicle) crosses the screen, but actually, there are often a plurality of moving objects simultaneously on the same screen. In this way, when a plurality of moving objects are present on one frame of the original image, a plurality of blob inclusion figures are also extracted, so that the blob inclusion figures extracted based on the same moving object are converted into a series of continuous blob inclusion figures. Need to be arranged on the time axis.

  FIG. 8 is a plan view showing an original image P10 in which a plurality of moving objects are present. This example is not a very realistic scenery, but for convenience of explanation, there are four types of moving objects, namely, a moving object O1 (horse), a moving object O2 (person), a moving object O3 (car), and a moving object O4 (bird) at the same time. It has become. Here, it is assumed that the horse O1 and the person O2 are moving from the left to the right of the screen, and the car O3 and the bird O4 are moving from the right to the left of the screen. Actually, a background image (city view) is shown in the back of these four types of moving objects O1 to O4. However, since the figure becomes complicated, each original image shown below has a background image. Drawing is omitted.

  FIG. 9 is a plan view showing a moving object identification image M10 created from the original image P10 shown in FIG. As described in §1, the moving body identification image M10 is a binary image created as a difference image between the original image P10 and the background image, and includes the background region K10 and the foreground regions F10a, F10b, F10c, and F10d. The

  Here, times t1, t2, t3,... Are defined on the time axis, and the original images obtained at these times are referred to as original images P1, P2, P3,. Try. The original image P10 shown in FIG. 8 is an original image obtained at time t10. FIG. 10 is a plan view showing original images P10, P20, P40 at times t10, t20, t40 on the time axis. That is, the original image P10 shown in FIG. 10A is an original image obtained at time t10. As described above, the horse O1 and the person O2 move to the right, and the car O3 and the bird O4 move to the left. ing. An original image P20 shown in FIG. 10B is an original image obtained at time t20, and each moving object continues to move in the traveling direction. The original image P40 shown in FIG. 10 (c) is the original image obtained at time t40, and the horse O1, the person O2, and the car O3 continue to move in the traveling direction, and the bird O4 has already left the screen. The dog O5 has appeared as a new moving object.

  11A to 11C show moving object identification images M10, M20, and M40 created from the original images P10, P20, and P40 shown in FIGS. 10A to 10C and blob inclusion figures extracted from these images. FIG. Here, K10, K20, and K40 are background areas in the moving object identification images M10, M20, and M40. A hatching area indicated by a code prefixed with the letter F is a foreground area, a two-digit number following the letter F indicates a time, and a lower-case alphabet at the end is a symbol for distinguishing moving objects. . On the other hand, a bold rectangle indicated by a symbol prefixed with the letter B is a blob inclusion figure extracted as a circumscribed rectangle of the foreground area (blob) in which the letter B is replaced with the letter F.

  For example, the hatching area F10a shown in the lower left of the moving object identification image M10 in FIG. 11A is a foreground area (blob) corresponding to the moving object O1 (horse) on the original image P10 shown in FIG. A blob inclusion figure B10a indicated by a bold rectangle is a blob inclusion figure extracted as a circumscribed rectangle of the foreground area F10a. Eventually, a total of four blob inclusion figures B10a to B10d are extracted on the moving object identification image M10 shown in FIG. 11 (a), and a total of four blob inclusion figures B20a are extracted on the moving object identification image M20 shown in FIG. 11 (b). -B20d are extracted, and a total of four blob inclusion figures B40a-B40c, B40e are extracted on the moving object identification image M40 shown in FIG. 11 (c).

  In this way, when a plurality of moving objects exist on the same screen and a plurality of blob inclusion figures are extracted, the blob inclusion figures extracted based on the same moving object are time-series as a series of continuous blob inclusion figures. It is important to line up. Such a series of continuous blob inclusion figures is generally called a tracker. FIG. 12 is a diagram illustrating an example of a tracker created based on the moving object identification images M10, M20, and M40 illustrated in FIG. This figure is composed of a table with a time axis in the horizontal direction and six trackers arranged in the vertical direction. Corresponding times t1 to t40 are shown in the first row of the table, and original images P1 to P40 obtained at each time are shown in the second row of the table. On the other hand, T1 to T6 described in the leftmost column are names of individual trackers, and the substance of each tracker is information on the arrangement of the blob inclusion figures described in the horizontal direction of the table.

  For example, the entity of the first tracker T1 is information indicating the arrangement of blob inclusion figures such as..., B10a,..., B20a,. Here, the blob inclusion figure B10a is a circumscribed rectangle with respect to the horse outline at time t10, as shown in FIG. 11 (a), and the blob inclusion figure B20a is at time t20, as shown in FIG. 11 (b). A circumscribed rectangle for the horse outline, and the blob inclusion figure B40a is a circumscribed rectangle for the horse outline at time t40, as shown in FIG. 11 (c). Eventually, the tracker T1 is a time series of blob inclusion figures representing horses. Similarly, the tracker T2 is a person, T3 is a car, T4 is a bird, and T5 is a blob inclusion figure showing a dog, arranged in time series.

  In the table shown in FIG. 12, for convenience of illustration, blob inclusion figures for times other than times t10, t20, and t40 are omitted, but of course, the blob inclusion figure for each original image of each frame is shown. It is extracted and arranged on a predetermined tracker. For example, in the case of a moving image composed of 30 frames of captured images per second, a blob inclusion figure is extracted for each original image obtained at each time every 1/30 seconds and arranged on the tracker.

  The tracker information as exemplified in FIG. 12 is nothing but the information indicating the result of moving body tracking. For example, the tracker T1 is the result of tracking a horse. From the left side of the table to the right side, following each blob inclusion figure ..., B10a, ..., B20a, ..., B40a, Can track the horse. More specifically, the rectangles constituting the blob inclusion figures B10a, ..., B20a, ..., B40a ... from the original images P10, ..., P20, ..., P40, ... If the inside is cut out and enlarged, the horse can be tracked in a zoomed up state.

  In the case of the embodiment shown here, one blob inclusion figure indicates a circumscribed rectangle of the contour of the moving object, and the substance of the data indicating the blob inclusion figure is the coordinates of two vertices located at the diagonal of the circumscribed rectangle Consists of values. FIG. 13 is a plan view showing entity data of the blob inclusion figure B. FIG. The illustrated example is an example in which the blob inclusion figure B is extracted as a circumscribed rectangle of the foreground area F (blob) corresponding to the moving object O3 (car). The blob inclusion figure B is an XY defined on the original image P. Using the coordinate system, the coordinate value (x1, y1) of the upper left vertex Q1 (x1, y1) and the coordinate value (x2, y2) of the lower right vertex Q2 (x2, y2) can be defined. (Of course, it may be defined by the lower left vertex and the upper right vertex). Therefore, each column of the tracker shown in FIG. 12 actually contains data indicating coordinate values such as “x1, y1, x2, y2”.

  In the case of the moving image illustrated in FIG. 10, the horse O1, the person O2, and the car O3 remain on the screen through the original images P10 to P40, but the bird O4 has left the screen before reaching the original image P40. The dog appears in the original image P40. Therefore, in the result of the moving body tracking shown in FIG. 12, a continuous blob inclusion figure array is present on the tracker T1 (horse), the tracker T2 (person), and the tracker T3 (car) even after the time t10 to t40. Although it exists, on the tracker T4 (bird), the continuous blob inclusion graphic sequence disappears immediately before time t40, and on the tracker T5 (dog), the continuous blob inclusion graphic sequence from time t40. Has begun. The thick and narrow ellipses shown in each tracker column in the figure indicate a series of continuous blob inclusion figures arranged along such a time series. Further, the column marked with X indicates that no blob inclusion figure is registered at the time. In other words, the presence of the column marked with “X” ends a series of continuous blob inclusion figure arrangements (that is, tracking of a certain moving object).

  When each blob inclusion figure shown in FIG. 11 is registered on the tracker shown in FIG. 12, the blob inclusion figure estimated to be extracted based on the same moving object is registered on the same tracker. There is a need. For example, when the blob inclusion figure B10a (horse) shown in FIG. 11 (a) is registered in the time t10 column of the tracker T1 as shown in FIG. 12, the blob inclusion figure B20a (horse) shown in FIG. 11 (b). 12 is registered in the time t20 column of the same tracker T1 as shown in FIG. 12, and the blob inclusion figure B40a (horse) shown in FIG. 11 (c) is also time t40 of the same tracker T1 as shown in FIG. Must be registered in the field.

  Here, if a human visually observes, it can be recognized that each original image in the blob inclusion figure B10a, B20a, B40a contains the same moving object called a horse. It is necessary to analyze the characteristics of moving objects using complex algorithms. Therefore, it is usually estimated that a blob inclusion figure containing the same moving object is estimated based on information such as the position, area, and shape of the blob inclusion figure and extracted based on the same moving object. Process to register blob inclusion figure on the same tracker.

  For example, based on the original image P10, as shown in FIG. 12, in the state where the blob inclusion figures B10a, B10b, B10c, and B10d are registered in the trackers T1, T2, T3, and T4, respectively, these blob inclusion figures are followed. Then, based on the original image P20, consider the process of registering the blob inclusion figures B20a, B20b, B20c, and B20d in any of the trackers (in practice, based on the original images P11 to P19 in the meantime). Blob inclusion figure is registered). In this case, among the four candidate blob inclusion figures shown in FIG. 11 (b), the most suitable blob inclusion figure to be registered in the tracker T1 following the blob inclusion figure B10a shown in FIG. The blob inclusion figure B20a having the most similar position, area, and shape is selected. Thus, as shown in FIG. 12, the blob inclusion figure B20a is registered in the time t20 column of the tracker T1.

  In short, even if the entity of the blob inclusion figure is a simple rectangular figure, the blob inclusion figure with similar position, area, and shape will be tracked in time series, and this will be registered on the same tracker. By adopting a simple method, a correct moving object tracking result can be obtained as shown in FIG. In such a method, it is not necessary to perform complicated processing for analyzing the characteristics of individual moving objects such as horses, people, cars, and birds.

<<< §3. Problems when multiple moving objects intersect >>>
As described above, if the technique of tracking the blob inclusion figure (the circumscribed rectangle in the embodiment shown here) in time series is used, the moving object tracking can be performed without performing complicated processing for analyzing the characteristics of the individual moving objects. It becomes possible. In fact, the process of quantitatively evaluating the similarity of the position, area, and shape of two rectangles can be performed by a relatively simple arithmetic operation process, so the process of tracking the blob inclusion figure in time series The computational burden is small. This is an advantage of the moving object tracking method using the blob inclusion figure.

  However, since a plurality of moving objects existing in the same screen are moving in arbitrary directions, there are cases where the movement paths intersect on the screen. For convenience of explanation, the three original images P10, P20, and P40 shown in FIGS. 10A, 10B, and 10C are obtained by avoiding a state in which a plurality of moving objects intersect with each other. However, in reality, a state occurs in which a plurality of moving objects overlap on the screen at these intermediate points in time.

  FIGS. 14A, 14B, and 14C are plan views showing original images P20, P30, and P40 at times t20, t30, and t40 in time series. Here, the original images P20 and P40 are exactly the same as the original images P20 and P40 shown in FIG. 10, but the original image P30 shows the state at the intermediate point. In the original image P30, the horse O1, the person O2, and the car O3 partially overlap, and only the bird O4 is spared.

  In the method described in §2, if the moving bodies overlap with each other due to the intersection of the moving paths of the plurality of moving bodies, accurate tracking cannot be performed. This is because, as shown as an original image P30 in FIG. 14 (b), a plurality of objects are merged into a group of objects at the time of intersection, and then separated into a plurality of original objects. Of course, if a human observes the original image P30, it is still possible to grasp the moving objects of the horse O1, the person O2, and the car O3 as individual moving objects, but the moving object using the blob inclusion figure described in §2 If the tracking method is adopted, the blob inclusion figure is formed in a state where the horse O1, the person O2, and the vehicle O3 are fused, so that individual moving objects cannot be recognized individually.

  FIGS. 15 (a), (b) and (c) show moving object identification images M20, M30 and M40 created from the original images P20, P30 and P40 shown in FIGS. 14 (a), (b) and (c). It is a top view which shows the blob inclusion figure extracted from these. Here, the moving body identification images M20 and M40 are exactly the same as the moving body identification images M20 and M40 shown in FIG. On the other hand, the moving object identification image M30 shows the state at the intermediate point, and is created based on the original image P30 shown in FIG. As shown in the drawing, only two blob inclusion figures B30d and B30g are extracted from the moving object identification image M30. Here, the blob inclusion figure B30d includes the outline of the bird O4, while the blob inclusion figure B30g includes the outline of the fusion of the horse O1, the person O2, and the car O3.

  In the moving body tracking method described in §2, the blob inclusion figure B30g cannot be recognized as including the outline of the fusion of horse O1, person O2, and car O3. Recognition will occur, and correct moving body tracking cannot be performed. What kind of misrecognition occurs depends on the specific method used to estimate the blob inclusion figure that contains the same moving object, based on information such as the position, area, and shape of the blob inclusion figure. is there.

  FIG. 16 is a diagram illustrating an example of a tracker created based on the moving object identification images M20, M30, and M40 illustrated in FIG. In this example, the blob inclusion figure B30g (human horse) is misrecognized as a car. That is, the blob inclusion figure B30g (human horse) shown in FIG. 15 (b) is suitable as the subsequent blob inclusion figure of the blob inclusion figure B20c (vehicle) shown in FIG. 15 (a) (ie, position, area, The shape is similar) and the blob inclusion figure B30g (carriage horse) is registered in the time t30 column of the tracker T3. On the other hand, the blob inclusion figure B30d (bird) shown in FIG. 15 (b) is determined to be suitable as the subsequent blob inclusion figure of the blob inclusion figure B20d (bird) shown in FIG. 15 (a) (this determination is correct). ), The blob inclusion figure B30d (bird) is registered in the time t30 column of the tracker T4. Since there is no other blob inclusion figure in the moving body identification image M30, there is no blob inclusion figure registered in the time t30 column of the trackers T1, T2, T5, and T6, and these columns are not registered (× mark). It has become.

  On the other hand, at time t40, the fusion state of the horse O1, the person O2, and the car O3 is canceled. For this reason, four blob inclusion figures B40a, B40b, B40c, and B40e are extracted from the moving object identification image M40. In the case of the example shown in FIG. 16, among the four blob inclusion figures extracted in this way, the blob inclusion figure B40b (person) is suitable as the subsequent blob inclusion figure of the blob inclusion figure B30g (human horse) (that is, the position). The blob inclusion figure B40b (person) is registered in the time t40 column of the tracker T3. Since the bird O4 has left the screen, the time t40 column of the tracker T4 is in an unregistered state (x mark).

  The remaining three blob inclusion figures B40a (horse), B40c (car), and B40e (dog) are new blob inclusion figures that appeared at time t40 (blob inclusion figures for new moving bodies that first appeared at time t40) ) And are registered in the time t40 column of trackers T1, T2 and T5 in the empty state. Here, the recognition that the blob inclusion figure B40e (dog) is a new blob inclusion figure is correct, but the blob inclusion figure B40a (horse) and B40c (car) are new blob inclusion figures. Recognition is wrong.

  If an incorrect moving object tracking result as shown in FIG. 16 is obtained, it is not possible to correctly perform processing for tracking and displaying a specific moving object in a zoomed-up state. For example, in order to zoom in and display the car O3, the blob inclusion figures B20c (car),..., B30g (carriage horse), ..., B40b (person) arranged on the tracker T3. If the inside of the rectangle that constitutes. Is cut out and enlarged, the tracking target to be enlarged will be mistaken by the person on the way from the car. When the horse O1 is tracked based on the tracker T1 or the person O2 is tracked based on the tracker T2, the tracking ends immediately before time t30.

  FIG. 17 is a diagram illustrating another example of the tracker created based on the moving object identification images M20, M30, and M40 illustrated in FIG. In this example, the blob inclusion figure B30g (human carriage) is misrecognized as a new participant. That is, the blob inclusion figure B30g (human horse) shown in FIG. 15 (b) is replaced with the blob inclusion figure B20a (horse), B20b (person), B20c (car), and B20d (bird) shown in FIG. 15 (a). It is determined that it is not suitable for any subsequent blob inclusion graphic (that is, the position, area, and shape are not similar), and the blob inclusion graphic B30g (cart) is registered in the time t30 column of the tracker T5. . In other words, the blob inclusion figure B30g (human horse) is determined to be a new participant that is neither a horse, a person, a car nor a bird. On the other hand, the blob inclusion figure B30d (bird) shown in FIG. 15 (b) is determined to be suitable as the subsequent blob inclusion figure of the blob inclusion figure B20d (bird) shown in FIG. 15 (a) (this determination is correct). ), The blob inclusion figure B30d (bird) is registered in the time t30 column of the tracker T4. Since there is no other blob inclusion figure in the moving body identification image M30, there is no blob inclusion figure registered in the time t30 column of the trackers T1, T2, T3, and T6, and these columns are not registered (× mark). It has become.

  On the other hand, at time t40, the fusion state of the horse O1, the person O2, and the car O3 is canceled. For this reason, four blob inclusion figures B40a, B40b, B40c, and B40e are extracted from the moving object identification image M40. In the case of the example shown in FIG. 17, all of the four blob inclusion figures extracted in this way are recognized as new blob inclusion figures that appear at time t40, and the times of trackers T1, T2, T3, and T6 in the empty state, respectively. It is registered in the t40 column. Here, the recognition that the blob inclusion figure B40e (dog) is the new blob inclusion figure is correct, but the blob inclusion figure B40a (horse), B40b (person), and B40c (car) are the new blob inclusion figures. Recognition of a figure is an error.

  If an incorrect moving body tracking result as shown in FIG. 17 is obtained, it is impossible to correctly perform processing for tracking and displaying a specific moving body in a zoomed up state. For example, when the horse O1 is tracked based on the tracker T1, the person O2 is tracked based on the tracker T2, and the car O3 is tracked based on the tracker T3, the tracking ends immediately before time t30. End up.

  In Patent Document 7 described above, as a technique for solving such a problem, when tracking a plurality of moving objects, it is determined whether or not the moving objects intersect with each other, and it is determined that they intersect. In such a case, a method for recognizing the depth relationship between a plurality of overlapping moving objects and performing correct tracking is disclosed. However, a complicated process is required for the process of determining the intersection of moving objects and the process of grasping the depth relationship, and it is difficult to always perform an accurate process in various situations. The present invention proposes a new approach capable of tracking individual moving objects efficiently and accurately even when a plurality of moving objects intersect. In the following, this new approach according to the present invention will be described in detail.

<<< §4. Recognition method of subsequent blob inclusion figure according to the present invention >>>
In §4, a case where the method for recognizing subsequent blob inclusion figures according to the present invention is applied to the case shown in FIG. 14 described in §3 will be described. The problem of this example is that, as shown in FIG. 14 (b), since a plurality of moving objects intersect on the original image P30 at time t30, as shown in FIG. 15 (b), from the moving object identification image M30. A blob inclusion figure B30g for a fusion of a plurality of moving objects is extracted. In the method for recognizing subsequent blob inclusion figures according to the present invention, it is possible to extract independent blob inclusion figures for each moving object even when a plurality of moving objects intersect as in the original image P30.

  Therefore, as shown in FIG. 18, when the transition from the original image P29 at time t29 to the original image P30 at time t30 occurs, the subsequent blob inclusion graphic for the blob inclusion graphic extracted from the original image P29 is shown below. A method for recognizing from the original image P30 will be described. However, here, for convenience of explanation, let us first consider a hypothetical example based on the following two points.

  The first premise is that a separate blob inclusion figure has been extracted for each of the four moving objects O1 to O4 from the original image P29 shown in FIG. 18 (a). As shown in the figure, since the moving objects O1 to O3 have already overlapped on the original image P29, one synthetic blob inclusion figure is extracted for the fusion of the moving objects O1 to O3 by the method described in §3. However, here, for each of the four moving objects O1 to O4, independent blob inclusion figures are extracted by some technique, and these blob inclusion figures are registered in the time t29 column of four different trackers. Assuming that

  The second premise is that the original image P29 shown in FIG. 18 (a) and the original image P30 shown in FIG. 18 (b) are exactly the same image. Actually, there is a slight time difference between the time t29 and the time t30, and each moving object has moved by a distance corresponding to the time difference in each traveling direction. Therefore, the original image P29 and the original image P30 are slightly different images. However, here, it is assumed that there is no movement of each moving body between time t29 and time t30, and both images are exactly the same image.

  19 (a) and 19 (b) are plan views showing moving object identification images M29 and M30 created from the original images P29 and P30 shown in FIGS. 18 (a) and 18 (b) and a blob inclusion figure extracted from them. It is. As shown in FIG. 19 (a), four independent blob inclusion figures B29a (horse), B29b (person), B29c (car), and B29d (bird) are extracted from the moving object identification image M29. These blob inclusion figures are registered in the time t29 column of trackers T1, T2, T3, and T4, respectively. The fact that independent blob inclusion figures are extracted despite the fact that humans, horses, and cars overlap to form a fusion is based on the first assumption described above.

  On the other hand, FIG. 19 (b) shows a state (the same state as FIG. 15 (b)) in which blob inclusion figures B30d and B30g are extracted from the moving object identification image M30 by the method described in §3. Has been. In this state, the subsequent blob inclusion figures (each) of the blob inclusion figures B29a (horse), B29b (person), B29c (car), and B29d (bird) registered in the time t29 column of the trackers T1, T2, T3, and T4 ( Even if an attempt is made to recognize a blob inclusion figure) to be registered in the time t30 column, correct recognition cannot be performed as described above. The reason is that the blob inclusion figure that should follow each blob inclusion figure B29a (horse), B29b (person), B29c (car) registered in the time t29 column of the tracker T1, T2, T3 is one blob inclusion. This is because the figure B30g has been merged. Therefore, for the blob inclusion figure B29d (bird) registered in the tracker T4, a subsequent blob inclusion figure B30d (bird) can be registered, but for other blob inclusion figures, Registration of blob inclusion figures will fail.

  Therefore, in the present invention, attention is paid to a certain moving object among a plurality of moving objects, and when the subsequent blob inclusion graphic for the blob inclusion graphic of this moving object is recognized, the preceding blob inclusion for moving objects other than the target moving object is recognized. A technique is used in which a moving object identification image in which a figure area is masked is used. Here, first, the processing procedure for recognizing the subsequent blob inclusion figure with the moving object O1 (horse) as the target moving object will be described. In other words, when the blob inclusion figure B29a (horse) shown in FIG. 19 (a) is registered in the time t29 field of the tracker T1, the subsequent blob inclusion figure to be registered in the time t30 field of the tracker T1 is recognized. Will be processed.

  For this purpose, first, a masking process is performed on the moving object identification image M30 shown in FIG. FIG. 20A is a plan view showing a moving object identification image obtained by performing such masking processing. Here, the moving object identification image after masking is referred to as an individual moving object identification image M (30, 1). In the figure, the hatched areas with double diagonal lines are rectangular areas corresponding to each blob inclusion figure, and moving objects other than the moving object O1 (horse) of interest by masking with these rectangular areas. Will be concealed. That is, the moving object O2 (person) is concealed by the blob inclusion figure B29b registered in the time t29 field of the tracker T2, and the moving object O3 (car) is concealed by the blob inclusion figure B29c registered in the time t29 field of the tracker T3. Then, the moving object O4 (bird) is concealed by the blob inclusion figure B29d registered in the time t29 column of the tracker T4. Eventually, only the target moving object O1 (horse) remains in the individual moving object identification image M (30, 1) obtained by the masking process.

  In FIG. 20 (a), for convenience of explanation, a mask area (area hidden by the masking process) is clearly shown as a rectangular area hatched by double diagonal lines. Is a part of the background area, and is originally an area to be shown as a white area. In FIG. 21 and later to be described later, for the convenience of explanation, the mask area is clearly shown as a hatched area with double diagonal lines. However, since each of the areas actually constitutes a part of the background area, This is an area to be shown as an area.

  FIG. 20 (b) is a plan view showing a blob inclusion figure B30a (bold line rectangle) extracted from the individual moving object identification image M (30, 1) shown in FIG. 20 (a). As described above, in the basic embodiment described here, the circumscribed rectangle of the foreground region is extracted as the blob inclusion graphic, and thus the blob inclusion graphic B30a is a circumscribed rectangle of the contour line of the moving object O1 (horse) of interest. Since this blob inclusion figure B30a is a candidate to be registered in the time t30 column of the tracker T1, it will be referred to as a candidate blob inclusion figure here. In this example, since the candidate blob inclusion figure extracted from the individual moving body identification image M (30, 1) is only the candidate blob inclusion figure B30a shown in FIG. 20 (b), the time t30 column of the tracker T1 includes This candidate blob inclusion graphic B30a is registered as a subsequent blob inclusion graphic. When there are a plurality of candidate blob inclusion figures, as described in §2, the one having the most similar position, area, and shape is selected from the candidate blob inclusion figures as the subsequent blob inclusion figure.

  Note that, due to the masking process described above, a part of the target moving object O1 (horse) is missing on the individual moving object identification image M (30, 1). In the illustrated example, a part of the neck of the horse happens to be missing, so the blob inclusion figure B30a is still a circumscribed rectangle of the outline of the moving object O1 (horse) of interest. If the portion is missing, a rectangle having a slightly smaller width than the original circumscribed rectangle is extracted as a blob inclusion figure. As described above, the blob inclusion figure extracted from the individual moving object identification image after the masking process may be slightly different from the circumscribed rectangle of the contour line of the original target moving object. Then there will be no problem.

  In the process in which the moving object O2 (person) is the target moving object and the subsequent blob inclusion figure is recognized, that is, the blob inclusion figure B29b (person) shown in FIG. 19A is registered in the time t29 column of the tracker T2. The process of recognizing the subsequent blob inclusion figure to be registered in the time t30 field can be performed in the same manner. Specifically, a masking process using a blob inclusion figure other than the target moving object (person) may be performed on the moving object identification image M30 shown in FIG. FIG. 21A is a plan view showing an individual moving body identification image M (30, 2) obtained by performing such masking processing. As shown in the figure, the moving object O1 (horse) is concealed by the blob inclusion figure B29a registered in the time t29 column of the tracker T1, and the moving object O3 (car) by the blob inclusion figure B29c registered in the time t29 column of the tracker T3. Is hidden, and the moving object O4 (bird) is concealed by the blob inclusion figure B29d registered in the time t29 column of the tracker T4, and only the moving object O2 (person) of interest is included in the individual moving object identification image M (30, 2). Will remain.

  FIG. 21 (b) is a plan view showing a candidate blob inclusion figure B30b (thick rectangle) extracted from the individual moving object identification image M (30, 2) shown in FIG. 21 (a). The candidate blob inclusion graphic B30b is registered as a subsequent blob inclusion graphic in the time t30 column of the tracker T2. Also here, due to the masking process, a part of the target moving object O2 (person) is missing on the individual moving object identification image M (30, 2). Specifically, since the lower body of the target moving object O2 (person) is missing, the blob inclusion figure B30b is a circumscribed rectangle of the outline of the upper body. However, as can be seen from the original image P30 shown in FIG. 18B, since the image information of the lower body of the moving object O2 (person) is originally missing on the original image P30, the blob inclusion figure B30b only shows the upper body. It does not cause any trouble even if it is shown.

  Similarly, when the process of recognizing the subsequent blob inclusion figure is performed with the moving object O3 (car) as the target moving object, the blobs other than the target moving object (car) are displayed on the moving object identification image M30 shown in FIG. What is necessary is just to perform the masking process using an inclusion figure. FIG. 22A is a plan view showing an individual moving body identification image M (30, 3) obtained by performing such masking processing. As illustrated, the moving object O1 (horse) is concealed by the blob inclusion figure B29a registered in the time t29 column of the tracker T1, and the moving object O2 (person) is concealed by the blob inclusion figure B29b registered in the time t29 column of the tracker T2. Is hidden and the moving object O4 (bird) is concealed by the blob inclusion figure B29d registered in the time t29 column of the tracker T4, and only the moving object O3 (car) of interest is included in the individual moving object identification image M (30, 3). Will remain.

  FIG. 22 (b) is a plan view showing a candidate blob inclusion figure B30c (thick line rectangle) extracted from the individual moving object identification image M (30, 3) shown in FIG. 22 (a). The candidate blob inclusion figure B30c is registered as a subsequent blob inclusion figure in the time t30 column of the tracker T3. In this case as well, due to the masking process, a part of the target moving object O3 (car) is missing on the individual moving object identification image M (30, 3), but there is no problem in tracking the moving object.

  Finally, the process of recognizing the subsequent blob inclusion figure is performed using the moving object O4 (bird) as the target moving object. The moving object O4 (bird) is a moving object that always maintains independence without merging with other moving objects, but in performing the process of recognizing the subsequent blob inclusion figure, The exact same approach is taken. That is, by performing a masking process using a blob inclusion figure other than the target moving object (bird) on the moving object identification image M30 shown in FIG. 19B, the individual moving object identification image as shown in FIG. Obtain M (30,4). As illustrated, the moving object O1 (horse) is concealed by the blob inclusion figure B29a registered in the time t29 column of the tracker T1, and the moving object O2 (person) is concealed by the blob inclusion figure B29b registered in the time t29 column of the tracker T2. Is hidden, and the moving object O3 (car) is concealed by the blob inclusion figure B29c registered in the time t29 column of the tracker T3, and only the moving object O4 (bird) of interest is included in the individual moving object identification image M (30, 4). Will remain.

  FIG. 23 (b) is a plan view showing a candidate blob inclusion figure B30d (bold line rectangle) extracted from the individual moving object identification image M (30, 4) shown in FIG. 23 (a). The candidate blob inclusion figure B30d is registered as a subsequent blob inclusion figure in the time t30 column of the tracker T4.

  FIG. 24 is a diagram showing a state in which subsequent blob inclusion figures are registered after the time t30 column of each tracker based on the candidate blob inclusion figures shown in FIGS. Here, the blob inclusion figures registered in the time t29 column are the blob inclusion figures B29a to B20d shown in FIG. 19 (a). Recognized subsequent blob inclusion figure.

  That is, for the tracker T1, when recognizing the subsequent blob inclusion graphic (the blob inclusion graphic to be registered in the time t30 column) for the blob inclusion graphic B29a (horse), the tracker T1 is used as the target tracker (that is, the horse is the target moving object). And) using the blob inclusion figures B29b (people), B29c (cars), and B29d (birds) registered in the time t29 field of the trackers T2, T3, and T4 that are not of interest. 20 (a), an individual moving object identification image M (30, 1) is generated, and a candidate blob inclusion figure B30a (horse) as shown in FIG. It will be recognized as a figure. FIG. 24 shows a state where the candidate blob inclusion figure B30a (horse) recognized as the subsequent blob inclusion figure is registered in the time t30 column of the tracker T1 of interest.

  When the subsequent blob inclusion figure for the blob inclusion figure B29b (person) is recognized for the tracker T2, this time, the tracker T2 is used as the notice tracker (that is, the person is the noticed moving object), and the non-target trackers T1, T3, Using the blob inclusion figures B29a (horse), B29c (car), and B29d (bird) registered in the time t29 column of T4, a masking process is performed on the moving body identification image M30, as shown in FIG. An individual moving object identification image M (30, 2) is generated, and a candidate blob inclusion figure B30b (person) as shown in FIG. 21 (b) is extracted and recognized as a subsequent blob inclusion figure. The subsequent blob inclusion graphic recognition processing for the trackers T3 and T4 is exactly the same.

  FIG. 25 is a plan view showing a state in which the subsequent blob inclusion figures B30a to B30d recognized for the trackers T1 to T4 are superimposed on the moving object identification image M30. Eventually, when the subsequent blob inclusion figure recognition method according to the present invention is adopted, instead of extracting the blob inclusion figures B30d and B30g shown in FIG. 19 (b) from the moving object identification image M30, the blob as shown in FIG. The inclusion figures B30a to B30d are extracted. In other words, even if a fusion object is formed on the original image P30 because a plurality of moving objects intersect, the contour of each moving object is included instead of the blob inclusion figure B30g that includes the outline of the fusion object. The blob inclusion figures B30a, B30b, and B30c are individually extracted and registered as subsequent blob inclusion figures of the respective trackers.

  As described above, taking the transition from time t29 to time t30 as an example, the procedure for recognizing the subsequent blob inclusion figure registered in the time t30 column of each tracker has been described, but the same procedure is executed for all the time columns. Become. As a result, as shown in FIG. 24, the tracker T1 records information including time series of blob inclusion figures for the moving object O1 (horse), and the tracker T2 stores the blob inclusion figure for the moving object O2 (person). Are recorded in time series, and information in which blob inclusion figures for the moving object O3 (car) are arranged in time series is recorded in the tracker T3. Actually, these three moving objects are overlapped before and after the time t30, but according to the above-described procedure, the blob inclusion for each moving object is obtained even during the overlapping period. The figures are extracted individually and registered in the correct tracker.

  If the table of FIG. 24 is compared with the tables of FIGS. 16 and 17, it can be easily understood that correct moving body tracking is performed. For example, in order to zoom in on the car O3 and display it in a tracking manner, a correct result of tracking the car O3 can always be obtained by cutting out and enlarging the inside of the rectangles constituting the blob inclusion figures arranged in the tracker T3.

  As described above, when the subsequent blob inclusion figure is registered in each tracker by the recognition method of the subsequent blob inclusion figure according to the present invention, the reason for obtaining a good moving body tracking result is different for each individual tracker. This is because a moving object identification image (an image masked using the preceding blob inclusion graphic of another tracker) is prepared, and a candidate blob inclusion graphic is extracted from the individual moving object identification image.

  That is, in the method described in §2 and §3, candidate blob inclusion figures B30d and B30g are extracted from the common moving object identification image M30 as shown in FIG. 19 (b), so that the correct subsequent blob inclusion figure is recognized. Could not do. On the other hand, in the method according to the present invention, the subsequent blob inclusion figure of the tracker T1 is extracted from the masked individual moving object identification image M (30, 1) as shown in FIG. The subsequent blob inclusion graphic is extracted from the masked individual moving object identification image M (30, 2) as shown in FIG. 21 (a), and the subsequent blob inclusion graphic of the tracker T3 is as shown in FIG. 22 (a). Extracted from the masked individual moving object identification image M (30, 3), and the subsequent blob inclusion figure of the tracker T4 is extracted from the masked individual moving object identification image M (30, 4) as shown in FIG. Therefore, it is possible to recognize a correct subsequent blob inclusion figure.

  In addition, the example used so far is an example which assumed two assumptions as mentioned above. First, as a first premise, as shown in FIG. 19 (a), each of the four moving bodies O1 to O4 is individually separated by some technique even though some moving bodies overlap at time t29. It was assumed that independent blob inclusion figures were extracted and these blob inclusion figures were registered in the time t29 column of four different trackers. However, the premise is not a special precondition, and is a precondition that is necessarily satisfied if the method for recognizing subsequent blob inclusion figures according to the present invention is employed.

  That is, if there is no overlap between the moving objects at the initial time t1, even if the moving objects overlap after that, as long as the subsequent blob inclusion figure is recognized in the above-described procedure, the individual moving objects are separated. It will be registered as an independent blob inclusion figure. For example, in the example shown in FIG. 14, since overlapping of moving objects does not occur in the original image P20 at time t20, even if overlapping of moving objects occurs in the process up to time t30, the individual moving objects are individually independent. As shown in FIG. 19 (a), at time t29 shown in FIG. 19 (a), it is assumed that a separate independent blob inclusion figure is registered for each of the four moving bodies O1 to O4. Become. Therefore, this first premise is not a special premise, but is a premise that is naturally satisfied if the above procedure is adopted.

  Next, the second premise is that the original image P29 shown in FIG. 18A and the original image P30 shown in FIG. 18B are exactly the same image. This is a premise that all the moving bodies O1 to O4 are stationary during the period from the time t29 to the time t30. Such a premise is nothing but a convenience for facilitating the explanation of the basic principle of the present invention.

  On the individual moving body identification image M (30, 1) after the masking process shown in FIG. 20 (a), moving bodies (people, cars, birds) other than the horse that is the moving body of interest are represented by blob inclusion figures B29b, B29c, B29d. The reason why it is completely hidden is because of the above assumption. That is, since the blob inclusion figures B29b, B29c, and B29d are circumscribed rectangles extracted based on the positions of people, cars, and birds at time t29, if masking processing is performed on the moving object identification image M29 at time t29, FIG. As in the example shown in (a), people, cars and birds can be completely hidden. However, in practice, the example shown in FIG. 20 (a) is the result of performing the masking process on the moving object identification image M30 at the time t30. It is not possible.

  FIG. 26 is a plan view (FIG. (A)) showing a state where complete masking is not performed by the masking process shown in FIG. 20, and a plan view showing candidate blob inclusion figures extracted from this state (FIG. (B)). It is. Actually, since the moving objects O1 to O4 are also moving during the period from time t29 to t30, a part of the moving object protrudes from the mask as in the example shown in FIG. Configure a part of the foreground area. As a result, four candidate blob inclusion figures B30a-1 to B30a-4 are extracted from the individual moving object identification image M (30, 1) as shown in FIG.

  However, even when a plurality of candidate blob inclusion figures are extracted in this way, if the optimum candidate blob inclusion figure is selected as the subsequent blob inclusion figure from among them, the candidate blob inclusion figure B30a-1 will eventually become the subsequent blob inclusion figure. The figure is selected and registered in the time t30 column of the tracker T1. Specifically, an operation of selecting a candidate blob inclusion graphic having the most similar position, area, and shape as the subsequent blob inclusion graphic may be performed on the previous blob inclusion graphic B29a shown in FIG. 19 (a). The candidate blob inclusion figures B30a-2 to B30a-4 that have not been selected are discarded without being registered in the tracker.

  Thus, the second premise described above is a premise intentionally provided for convenience of explanation, and is not an essential precondition for executing the method for recognizing subsequent blob inclusion figures according to the present invention. Therefore, no special preconditions are necessary for carrying out the present invention.

<<< §5. Recognition method of new blob inclusion figure according to the present invention >>>
In §4, the basic principle of the method for recognizing subsequent blob inclusion figures according to the present invention has been described. This subsequent blob inclusion figure recognition process is a process for adding a new blob inclusion figure following the previously registered previous blob inclusion figure, and is continuously moving on the screen. It can be said that it is a process for continuing the tracking of the moving object.

  However, usually, the phenomenon that the moving object existing on the screen disappears in the meantime and the new moving object appears on the screen is repeated. Therefore, in practice, in addition to the process of tracking the moving object moving on the screen, it is necessary to perform the process when the existing moving object disappears and the process when the new moving object appears. Also in the case of the example shown in FIG. 14, the moving object O4 (bird) no longer exists in the original image P40 obtained at time t40, and a newcomer moving object O5 (dog) appears instead. As a result, in the table shown in FIG. 24, the series of blob inclusion figures for the bird of the tracker T4 has ended in the middle of reaching the time t40, and the time t40 column is in an unregistered state (x mark). ing. On the other hand, in the time t40 column of the tracker T5, a blob inclusion graphic B40e for the newcomer moving object O5 (dog) is registered as a new blob inclusion graphic.

  Here, the process when the existing moving object disappears can be executed as a process when the subsequent blob inclusion figure cannot be recognized in the procedure described in §4. For example, consider a case where the moving object O4 (bird) exists until time t38 (including a case where only a part exists near the frame of the screen) but jumps out of the screen at time t39. In this case, even if the process for recognizing the subsequent blob inclusion graphic to be registered in the time t39 field of the tracker T4 is performed, the candidate blob inclusion graphic suitable for the subsequent blob inclusion graphic cannot be recognized. Unregistered state (x mark). This is nothing but the process when the moving object O4 (bird) disappears. That is, on the tracker T4, a series of blob inclusion graphic groups that have been continuously registered in time series are interrupted by the last blob inclusion graphic in the time t38 column, and thereby the moving object O4 (bird) disappears. Information that it was done will be recorded.

  Note that the disappearance of the moving object is not necessarily limited to the case of moving out of the screen. For example, when a person enters a building or a car gets off to an underground parking lot, the moving object may disappear near the center of the screen. However, no matter where the moving object disappears, there is no change in the state that the following blob inclusion figure cannot be recognized on the tracker, and it is recorded as an unregistered state (x mark).

  On the other hand, when a new participant appears, it is necessary to recognize the new blob inclusion figure and perform a process of newly registering it in a predetermined tracker (any tracker in an empty state). Therefore, as shown in FIG. 27, when a transition from the original image P39 at time t39 to the original image P40 at time t40 occurs, a new blob inclusion figure about the new participant O5 that suddenly appeared in the original image P40. The procedure of the process of registering on the tracker will be described.

  28 (a) and 28 (b) are plan views showing moving object identification images M39 and M40 created from the original images P39 and P40 shown in FIGS. 27 (a) and 27 (b) and a blob inclusion figure extracted from them. It is. Here, based on the moving body identification image M39 shown in FIG. 28 (a), the blob inclusion figure B39a (horse), B39b (person), and B39c (car) are already registered in the tracker. An operation for registering a new blob inclusion graphic in the time t40 field based on the moving object identification image M40 shown in b) will be described.

  FIG. 29 is a diagram showing a tracker created based on the blob inclusion graphic shown in FIG. It is assumed that the blob inclusion figure B39a (horse), B39b (person), and B39c (car) are already registered in the trackers T1, T2, and T3 in the time t39 column, respectively. On the other hand, in the time t40 column, first, the recognition method of the subsequent blob inclusion figure described in §4 is applied to register the blob inclusion figure B40a (horse), B40b (person), and B40c (car). In the registration, the individual moving object identification images M (40,1), M (40,2), M (40,3) obtained by performing the masking process on the moving object identification image M40 shown in FIG. ).

  Specifically, the individual moving object identification image M (40, 1) is obtained from the moving object identification image M40 by the masking process using the blob inclusion figure B39b (person) and B39c (car), and extracted from here. The blob inclusion figure B40a (horse) is registered in the tracker T1. Similarly, the blob inclusion figure B40b (person) extracted from the individual moving object identification image M (40, 2) obtained by the masking process using the blob inclusion figure B39a (horse) and B39c (car) is registered in the tracker T2. The blob inclusion figure B40c (car) extracted from the individual moving object identification image M (40, 3) obtained by the masking process using the blob inclusion figure B39a (horse) and B39b (person) is registered in the tracker T3. The FIG. 29 shows a state where registration of these subsequent blob inclusion figures is completed.

  The process of recognizing a new participant from the original image P40 shown in FIG. 27 (b) and registering it in the tracker is performed after the registration of the subsequent blob inclusion figure is completed. The basic principle is registered in the blob inclusion figure B40a (horse), B40b (person), B40c (car), and B40e (dog) extracted from the moving object identification image M40 shown in FIG. It is determined whether there is a blob inclusion figure that does not correspond to the succeeding blob inclusion figure, and if such a blob inclusion figure exists, it is recognized as a new blob inclusion figure, and the time t40 column is not yet displayed. The first blob inclusion figure is registered in the registered tracker.

  Specifically, in the case of the above example, as shown in FIG. 29, the subsequent blob inclusion figures B40a (horse), B40b (person), and B40c (car) are already registered in the trackers T1, T2, and T3. Of the four blob inclusion figures extracted from the moving object identification image M40 shown in FIG. 28 (b), the blob inclusion figure that does not correspond to the subsequent registered blob inclusion figure is only the blob inclusion figure B40e (dog). . Therefore, this blob inclusion figure B40e (dog) is determined to be a new blob inclusion figure, and the time t40 column is registered as the first blob inclusion figure in the unregistered (blank state) tracker. Of course, if there are multiple new blob inclusion figures, each will be registered in a different tracker.

  In the case of the example shown in FIG. 29, since the tracker T4, T5, and T6 are all unregistered in the time t40 column, the new blob inclusion figure B40e (dog) may be registered in any of these. The example shown in FIG. 24 is an example in which this new blob inclusion figure B40e (dog) is registered in the tracker T5. The time t40 column of the remaining trackers T4 and T6 is in an unregistered state (x mark) because there is no blob inclusion figure to be registered. The newly added blob inclusion figure B40e (dog) registered in this way functions as a leading blob inclusion figure, and subsequent blob inclusion figures are arranged in the tracker T5 until the moving object O5 (dog) disappears thereafter.

  As shown in FIG. 27 (b), there are no moving objects that overlap each other on the original image P40. Therefore, blob inclusion figures B40a (horse) and B40b (people) extracted from the moving object identification image M40 shown in FIG. 28 (b). ), B40c (car) and the subsequent blob inclusion figures B40a (horse), B40b (person), and B40c (car) registered in the time t40 column in the table shown in FIG. 29 have a one-to-one correspondence. . For example, the blob inclusion figure B40a (horse) registered in the time t40 column of the tracker T1 in FIG. 29 is the individual moving object identification image M (40, 1) after the masking process for the moving object identification image M40 shown in FIG. The blob inclusion figure extracted from (1)) matches the blob inclusion figure B40a (horse) originally included in the moving object identification image M40. However, the blob inclusion figure B40e (dog) extracted from the moving object identification image M40 shown in FIG. 28B does not have a corresponding subsequent blob inclusion figure on any of the trackers shown in FIG. Therefore, the blob inclusion graphic B40e (dog) can be recognized as a new blob inclusion graphic.

  Note that when there are moving objects that overlap each other on the original image, a simple one-to-one correspondence as in the above example is not achieved. For example, since three moving objects O1, O2, and O3 overlap on the original image P30 shown in FIG. 18 (b), two blob inclusion figures as shown in FIG. 19 (b) are obtained from the moving object identification image M30. B30d (bird) and B30g (passenger horse) are extracted. On the other hand, in the table shown in FIG. 24, the subsequent blob inclusion figures B30a (horse), B30b (person), and B30c (car) registered in the time t30 column of the trackers T1, T2, and T3 are the individual after masking processing. Since it is extracted from the moving body identification images M (30,1), M (30,2), M (30,3), the blob inclusion figure B30g (human horse) shown in FIG. There is no one-to-one correspondence.

  Thus, when a plurality of n moving objects overlap with each other, one blob inclusion figure is extracted from the moving object identification image M for the fusion object of the plurality n moving objects. However, a one-to-one correspondence is not obtained for the subsequent blob inclusion figure registered in (1), but instead a one-to-n correspondence is obtained. That is, in the case of the above example, the three subsequent blob inclusion figures B30a registered in the time t30 column of the table of FIG. 29 with respect to the blob inclusion figure B30g (human horse) shown in FIG. 19 (b). A one-to-three correspondence relationship in which (horse), B30b (person), and B30c (car) correspond is obtained. On the other hand, for the blob inclusion graphic B30d (bird) in FIG. 19B, a one-to-one correspondence is obtained with the subsequent blob inclusion graphic B30d (bird) registered in the time t30 column of the table of FIG. .

  As described above, even when moving objects that overlap each other exist on the original image, there is a one-to-one correspondence between each blob inclusion graphic extracted from the moving object identification image M and the subsequent blob inclusion graphic registered on the tracker. Alternatively, a one-to-n correspondence is obtained. Therefore, as described above, it is possible to recognize a blob inclusion graphic whose corresponding subsequent blob inclusion graphic is not registered on the tracker as a new blob inclusion graphic, and newly register it on the tracker. In the case of the moving body identification image M30 shown in FIG. 19B, the extracted blob inclusion figure B30g (human horse) corresponds to the three subsequent blob inclusion figures B30a (horse), B30b (person), and B30c (vehicle). Since the extracted blob inclusion graphic B30d (bird) corresponds to a single subsequent blob inclusion graphic B30d (bird), it is determined that there is no new blob inclusion graphic on the moving object identification image M30. It can be performed.

  In addition, in order to contribute to the use of determining the presence or absence of such a new blob inclusion figure, in practice, when registering the subsequent blob inclusion figure on the tracker, the subsequent blob inclusion figure and the subsequent blob inclusion figure are extracted. It is preferable to create a table indicating the correspondence relationship with the blob inclusion figure on the moving object identification image M that is the basis of the above. FIG. 30 shows an example of such a table. This example is a table showing the correspondence relationship between the subsequent blob inclusion graphic registered in the column of time t30 in the table of FIG. 24 and the blob inclusion graphic on the moving object identification image M30 shown in FIG. It can be created when performing the work of registering each subsequent blob inclusion figure on the tracker.

  For example, consider the case where the subsequent blob inclusion figure B30a (horse) extracted from the individual moving object identification image M (30, 1) shown in FIG. 20B is registered in the time t30 column of the tracker T1 in the table of FIG. Like. In this case, the subsequent blob inclusion figure B30a (horse) shown in FIG. 20 (b) and the blob inclusion figure B30g (human horse) shown in FIG. 19 (b) have a relationship in which the latter includes the former. I understand that. In this way, the blob inclusion figures in which one of them includes the other are recognized as blob inclusion figures corresponding to each other, and the corresponding relation is recorded on the table. The correspondence recorded in the first row of the table shown in FIG. 30 is recorded in this way.

  Similarly, when the subsequent blob inclusion figure B30b (person) extracted from the individual moving body identification image M (30, 2) shown in FIG. 21B is registered in the time t30 column of the tracker T2 in the table of FIG. 24, when the subsequent blob inclusion figure B30c (car) extracted from the individual moving object identification image M (30, 3) shown in FIG. 22B is registered in the time t30 column of the tracker T3 in the table of FIG. The correspondence relationship with the blob inclusion figure B30g (human horse) in the inclusion relationship is recorded. The correspondence recorded in the second and third rows of the table shown in FIG. 30 is recorded in this way.

  Also, when registering the subsequent blob inclusion figure B30d (bird) extracted from the individual moving body identification image M (30, 4) shown in FIG. 23 (b) in the time t30 column of the tracker T4 in the table of FIG. 24, Since the subsequent blob inclusion graphic B30d (bird) and the blob inclusion graphic B30d (bird) shown in FIG. 19 (b) are in a matching relationship (one-to-one correspondence), the blob that corresponds to each other again. It is recognized as an inclusion figure, and the corresponding relationship is recorded on a table. The correspondence recorded in the fourth row of the table shown in FIG. 30 is recorded in this way.

  In short, when performing the process of registering each subsequent blob inclusion figure on the tracker, the subsequent blob inclusion figure to be registered is compared with each blob inclusion figure on the moving object identification image M, and they match. Alternatively, what is in a relationship including one of the other may be recognized as a blob inclusion figure corresponding to each other, and the corresponding relationship may be recorded in a table. If such a table is prepared, it is possible to easily determine whether or not a new blob inclusion graphic exists on the moving object identification image M as described above. That is, among the individual blob inclusion figures on the moving object identification image M, a blob inclusion figure in which the correspondence relationship with the subsequent blob inclusion figure is not recorded can be recognized as a new blob inclusion figure.

<<< §6. Moving body tracking device according to basic embodiment >>
The moving body tracking apparatus according to the present invention employs the subsequent blob inclusion graphic recognition method described in §4 and the new blob inclusion graphic recognition method described in §5 to register the new blob inclusion graphic on the tracker and The subsequent blob inclusion figure is registered, and the process of tracking individual moving objects is performed. The configuration and operation of the moving object tracking device according to the basic embodiment of the present invention will be described below with reference to the block diagram of FIG.

  This moving body tracking device is a device that detects a moving body from an input moving image and performs processing for tracking the moving body. Its basic role is to create and hold tracker information described so far. is there. As shown in the figure, the basic components of this apparatus are an image input unit 110, a background image providing unit 120, a moving object identification image generating unit 130, a masking processing unit 140, a tracker storage unit 150, a tracker registration unit 160, a blob inclusion figure. This is the extraction unit 170. Hereinafter, functions of these components will be described in order.

<6-1: Image Input Unit 110>
The image input unit 110 is a component that inputs a moving image composed of a plurality of original images P (i) continuous in time series and provides the moving image to the moving object identification image generation unit 130. Here, i is a natural number (i = 1, 2, 3,...) Indicating the order of time series. In §3, as shown in FIGS. 14 (a), (b), and (c), an explanation was given using an example in which original images P20, P30, and P40 were input at times t20, t30, and t40. Here, the original image P (1) is given to the image input unit 110 at time t (1), the original image P (2) is given at the next time t (2),. Assume that an original image P (i) is given at t (i). A plurality of original images P (i) continuous in time series thus provided constitute a moving image. For example, in the case of a moving image of 30 frames / second, the original image P (i + 1) is a frame 1/30 seconds after the original image (i).

  In addition, if the image input unit 110 is configured by a storage device such as a hard disk device, a moving image composed of a plurality of input original images P (i) can be accumulated as moving image data. An original image of a necessary frame can be provided to the moving object identification image generation unit 130. Therefore, it is possible to perform the moving body tracking process for any part of the moving image at any time. Further, if the original image is provided in reverse order so as to go back in time series to the moving object identification image generation unit 130, moving object tracking can be performed while moving the moving image backward. Of course, when used for the purpose of tracking a moving object on the spot based on a moving image captured in real time, it is not necessary to provide a function for accumulating and storing moving image in the image input unit 110.

<6-2: Background Image Providing Unit 120>
The background image providing unit 120 is a component having a function of providing the background image constituting the background of the moving image provided to the image input unit 110 to the moving object identification image generating unit 130. For example, if the background image providing unit 120 has a function of capturing and storing a still image given from the outside, and a predetermined still image is given as a background image in advance, the background image providing unit 120 The still image can be provided as a background image as it is. If the background image is provided by such a method, the background image providing unit 120 can be configured by a simple data storage device that stores still image data.

  Of course, as described above, the background image providing unit 120 can be configured by a simple data storage device when handling a moving image captured by a fixed point camera in a room where the lighting environment does not change. When dealing with moving images taken indoors or outdoors where the lighting environment changes from time to time due to external light, naturally the background image also needs to change over time. That is, even a moving image captured by a fixed point camera changes with the passage of time such as morning, noon, and evening, and the background image also changes due to weather such as movement of clouds and changes in sunshine conditions. The background image also changes when the location of the fixed point camera and the shooting direction are changed.

  In order to cope with such a change in the background image, the background image providing unit 120 has a function of obtaining a simple average or a weighted average of the original images input by the image input unit 110 over a predetermined period in the past. The average image obtained may be provided as a background image. The average image is suitable for use as a background image because the moving object information is diluted. The background image providing unit 120 of the embodiment shown in FIG. 31 is a component having such a function. That is, the original image P (i) input by the image input unit 110 is sequentially given to the background image providing unit 120. The background image providing unit 120 has a function of creating an average image based on the original image P (i) thus provided and providing the average image as a background image to the moving object identification image generating unit 130.

  The specific method for creating the background image by taking the average of the past original images is as described in section 1 above. For example, if an image indicating a simple average of original images obtained in the past 10 minutes (an image obtained by calculating a simple average of pixel values for each pixel) is used as a background image, the background image providing unit 120 includes: It is always necessary to incorporate a data storage location for accumulating the original image that has been traced back for 10 minutes from the present time, and a calculation function for performing an average calculation of pixel values. Of course, it is also possible to obtain a weighted average that gives higher weight to the pixels of the original image closer to the current time.

  However, in order to accumulate 10-minute image data of a moving image of about 30 frames / second, a considerable storage capacity is required. Therefore, as described in §1, in practice, the i-th original image P ( i), the i-th average image A (i-1) and the i-th original image P (i) are used to determine the i-th It is preferable to adopt a method of creating the th average image A (i). The specific method is as described in FIG.

<6-3: Moving Object Identification Image Generation Unit 130>
The moving object identification image generation unit 130 compares the i-th original image P (i) provided from the image input unit 110 with the background image provided from the background image providing unit 120, and compares the background region and the foreground region. And the i-th moving object identification image M (i) for distinguishing between them. In the case of the embodiment shown here, as described with reference to FIG. 2, the i-th original image P (i), the (i−1) -th average image (ie, the background image) A (i−1), , The i-th moving object identification image M (i) is generated. The moving object identification image M (i) is a binary image that distinguishes the background region K (i) and the foreground region F (i).

  Here, in order to generate the moving object identification image M (i), as described in FIG. 5, the color of the pixel p (i) at a specific position on the original image P (i) and the background image A (i− 1) Compare the color of the pixel a (i-1) at the specific position above, and if the degree of approximation of the colors of both pixels satisfies a predetermined condition, the pixel m (i) at the specific position If the pixel in the region K is not satisfied and the predetermined condition is not satisfied, the pixel m (i) at the specific position may be the pixel in the foreground region F. As the predetermined condition for the degree of approximation, for example, as described in §1, the Euclidean distance in the color space can be set.

<6-4: Tracker storage unit 150>
The tracker storage unit 150 is a component that stores tracker information for each moving object. Specifically, the tracker storage unit 150 has a function of storing table information as illustrated in FIGS. 24 and 29. Here, as described above, the tracker is information in which a plurality of blob inclusion figures estimated to be extracted based on the same moving object are arranged in time series from the first blob inclusion figure to the last blob inclusion figure. .

  In the block of the tracker storage unit 150 in FIG. 31, a table in which a plurality of trackers taking time series in the horizontal direction are arranged in the vertical direction is illustrated. Here, T1, T2,..., Tj,... Are names of individual trackers, and each tracker is provided with a column corresponding to a specific time, and a specific blob inclusion figure. Is registered. For example, looking at the column of the j-th tracker Tj, the blob inclusion figure Bj (i−1) is registered in the column of time t (i−1), and the blob in the column of time t (i). An inclusion figure Bj (i) is registered. Of course, there is an unregistered state column where the blob inclusion figure has not yet been registered, and there is an unregistered state column indicating that there is no blob inclusion figure to be registered.

  In the case of the embodiment shown here, as shown in FIG. 13, each blob inclusion figure B is constituted by a circumscribed rectangle of a moving object, and its substance is two vertices Q1 (x1, y1), Q2 (x2 , Y2). Therefore, the substance of information stored in each column of the tracker storage unit 150 is data of coordinate values indicating two vertices of a circumscribed rectangle such as (x1, y1, x2, y2). That is, the tracker storage unit 150 stores the coordinate values of the two vertices located at the diagonal of the circumscribed rectangle as tracker information.

  Of course, the tracker storage unit 150 only needs to have a function of storing a plurality of trackers indicating information obtained by arranging a series of blob inclusion figures in time series. In what form the information of the line is recorded is arbitrary. Therefore, the specific data structure stored in the tracker storage unit 150 is not limited to the form illustrated here.

<6-5: Masking processing unit 140>
The masking processing unit 140 is a component that performs masking using a specific blob inclusion figure stored in the tracker storage unit 150 on the foreground region F of the moving object identification image M generated by the moving object identification image generation unit 130. is there. More specifically, the jth tracker Tj stored in the tracker storage unit 150 is used as the target tracker, and the blob belonging to the (i-1) th time series of the non-target tracker (tracker other than Tj). The individual moving object identification image M (i, j) for the tracker Tj of interest is masked by masking the i-th moving object identification image M (i) using the rectangle that is the inclusion figure B (i-1). Is generated and provided to the blob inclusion figure extraction unit.

  For example, if j = 1, the tracker T1 becomes the target tracker, and all the trackers after the tracker T2 become the non-target trackers. Then, a masking process is performed on the moving object identification image M (i) using a rectangle that is the blob inclusion figure B (i-1) belonging to the (i-1) th time series of each non-target tracker. As a result, the individual moving object identification image M (i, 1) for the tracker T1 of interest is generated. Similarly, if J = 2, an individual moving object identification image M (i, 2) for the target tracker T2 is generated, and if J = 3, an individual moving object identification image M (i, 3) for the target tracker T3. ) Is generated.

  20 (b), 21 (b), 22 (b), and 23 (b), the individual moving body identification images M (30, 1), M (30, 2), M (30, 3) and M (30, 4) are actual examples of the individual moving object identification images created in this way. Since these individual moving body identification images are used for the subsequent blob inclusion graphic registration process in the tracker registration unit 160, it is not necessary to create a tracker that does not need to find the subsequent blob inclusion graphic.

  For example, in the case of the example shown in FIG. 24, for the trackers T1 to T4, the blob inclusion graphic is registered in the time t29 column, and it is necessary to find the subsequent blob inclusion graphic to be registered in the time t30 column. It is necessary to create individual moving body identification images M (30,1), M (30,2), M (30,3), and M (30,4) with T4 as the target tracker. For T6, since the time t29 column is in an unregistered state (x mark), there is no subsequent blob inclusion figure to be registered in the time t30 column. Therefore, it is not necessary to create the individual moving object identification images M (30, 5) and M (30, 6) using the trackers T5 and T6 as the target tracker. After all, if the masking processing unit 140 generates the individual moving object identification image M (i, j) only for the tracker in which the blob inclusion figure B (i-1) is registered in the (i-1) th time series. It ’s enough.

<6-6: Blob inclusion figure extraction unit 170>
The blob inclusion figure extraction unit 170 recognizes a group of pixels (blob) constituting the foreground region F as a moving object based on the i-th moving object identification image M (i) generated by the moving object identification image generation unit 130. And the function of extracting the circumscribed rectangle of the pixel group as a blob inclusion figure B (i) belonging to the i-th time series and the individual moving object identification image M (i, j) generated by the masking processing unit 140, A group of pixels (blob) constituting the foreground area F excluding the masked portion is recognized as a moving object, and the circumscribed rectangle of the pixel group recognized as a moving object is to belong to the i-th time series. And a function of extracting as a candidate blob inclusion figure for the tracker.

  Specifically, blob inclusion figures B30g and B30d (rectangles indicated by bold lines) shown in FIG. 19 (b) are blob inclusion figures extracted from the moving object identification image M30. FIG. 20 (b) and FIG. ), Blob inclusion figures B30a, B30b, B30c, and B30d shown in FIG. 22 (b) and FIG. 3) A blob inclusion figure extracted from M (30, 4).

  Note that several specific methods are known for extracting a circumscribed rectangle as a blob inclusion figure from a group of pixels constituting the foreground region F on the moving object identification image M. Here, a method based on a relatively simple algorithm is described below, taking as an example a case where the moving object identification image M is a general image composed of a pixel array in which small rectangular pixels are arranged in rows and columns. An example is given below. FIG. 32 is a flowchart showing an example of the procedure of the blob inclusion figure extraction process.

  First, in step S1, an initial target pixel is set on the moving object identification image M. When the moving object identification image M is a general image having vertical and horizontal pixel arrays, for example, the upper left pixel (the pixel in the first row and the first column) may be set as the initial target pixel. In subsequent step S2, it is determined whether or not the pixel of interest is a foreground pixel. For example, in the case of the moving object identification image M created by the method shown in FIG. 5, if the pixel value = 1, it is a foreground pixel.

  If it is determined in step S2 that the pixel is a foreground pixel, it is determined in step S3 whether or not a registered rectangle exists at a distance within n pixels of the pixel of interest. Here, as described later, the registered rectangle is the rectangle registered in step S4 or the rectangle expanded in step S5 based on the rectangle. On the other hand, if it is not determined in step S2 that the pixel is a foreground pixel (that is, if the target pixel is a background pixel), the process proceeds to step S6.

  Subsequently, a branch based on the determination result of step S3 will be described. First, in step S3, when it is determined that there is no registered rectangle at a distance within n pixels of the target pixel, in step S4, processing for registering the outline rectangle of the target pixel itself is performed. On the other hand, if it is determined in step S3 that the registered rectangle exists at a distance within n pixels of the target pixel, in step S5, the region of the registered rectangle is expanded to a circumscribed rectangle including the target pixel. Is done.

  If such processing is repeatedly executed through steps S6 and S7 until all the pixels are focused, that is, if scanning is performed with all the pixels as the focused pixels, this blob inclusion figure extraction processing is completed. The rectangle registered at the time of completion becomes a circumscribed rectangle to be extracted as a blob inclusion figure.

  Here, the process when the blob inclusion figure extraction process shown in FIG. 32 is performed with a specific image as shown in FIG. 33 as a processing target will be briefly described. FIG. 34 is a plan view showing each stage of the process to which the procedure of the blob inclusion figure extraction process is applied.

  The image illustrated in FIG. 33 is a simple image composed of a total of 72 pixels arranged in 6 rows and 12 columns. Here, it is assumed that the hatched pixels are foreground pixels and the white pixels are background pixels. As shown in the figure, this image includes two sets of a group of pixels (a group of foreground pixels with hatching: that is, a blob) constituting the foreground region, and correct blob inclusion figure extraction processing is executed. For example, circumscribed rectangles for these two sets of pixel groups (blobs) should be extracted as blob inclusion figures, respectively. Note that the pixel of interest is scanned sequentially from the left pixel to the right pixel included in the same row, and when the scanning of one row is completed, the scanning of the next row is started.

  First, in step S1, the pixel in the first row and the first column is set as the initial target pixel. Since the initial target pixel is not a foreground pixel, the pixel in the first row and the second column becomes the next target pixel through steps S2 to S6 and S7. Since the target pixel is not a foreground pixel, the pixel in the first row and the third column becomes the next target pixel. In the case of the example shown in FIG. 33, since there is no foreground pixel in the first row, the process focusing on the pixel in the second row is continued.

  Then, the pixel of interest becomes the foreground pixel only when the pixel in the second row and second column becomes the pixel of interest. Therefore, the determination in step S3 is performed. At this point, since there is no registered rectangle, the process proceeds to step S4, and the outline rectangle of the pixel of interest itself is registered as a registered rectangle. FIG. 34 (a) shows a state in which such registration has been made. A rectangle indicated by a thick line in the figure is a registered rectangle. Subsequently, through steps S6 and S7, the pixel in the second row and the third column becomes the next pixel of interest, but since the foreground pixel does not exist in the second row thereafter, the processing focusing on the pixel in the third row Will continue.

  Then, when the pixel in the third row and the second column becomes the target pixel, the target pixel becomes the foreground pixel again. Therefore, the determination in step S3 is performed. Here, for convenience of explanation, let us consider a case where n = 3 (practically, a larger n is generally set). Then, as shown in FIG. 34 (a), since there is a registered rectangle (thick rectangle) within a distance of 3 pixels or less with respect to the pixel of interest in the third row and the second column, the process proceeds to step S5, and the registered rectangle Is expanded to a circumscribed rectangle including the target pixel. FIG. 34 (b) shows a state where such an extension has been made. The registration rectangle, which has been only one pixel in size so far, has been expanded to a vertically long registration rectangle for two pixels.

  Subsequently, through steps S6 and S7, the pixel in the third row and the third column becomes the next pixel of interest, and this pixel is also a foreground pixel. Therefore, when the determination in step S3 is performed, as shown in FIG. 34 (b), there is a registered rectangle (thick rectangle) within a distance of 3 pixels or less for the pixel of interest in the third row and the third column. Proceeding to S5, processing for expanding the area of the registered rectangle to a circumscribed rectangle including the pixel of interest is performed. FIG. 34 (c) shows a state where such an extension has been made. The registration rectangle that has been only two pixels in size so far has been expanded to a registration rectangle for four pixels.

  Subsequently, through steps S6 and S7, the pixel in the third row and the fourth column becomes the next pixel of interest, but this pixel is not the foreground pixel. Therefore, through steps S6 and S7, the pixel in the third row and the fifth column becomes the next pixel of interest, and this pixel is a foreground pixel. Therefore, when the determination in step S3 is performed, as shown in FIG. 34 (c), there is a registered rectangle (thick line rectangle) at a distance within 3 pixels for the pixel of interest in the third row and the fifth column. Proceeding to S5, processing for expanding the area of the registered rectangle to a circumscribed rectangle including the pixel of interest is performed. FIG. 34 (d) shows a state where such an extension has been made. The registration rectangle that has been only 4 pixels in size so far has been expanded to a registration rectangle of 8 pixels.

  The foreground pixel becomes the target pixel again at the third row and the tenth column. When the pixel in the third row and the tenth column is the target pixel and the determination in step S3 is performed, as shown in FIG. 34 (d), although there are registered rectangles (thick rectangles), they are separated by 5 pixels. There is no registered rectangle with a distance within 3 pixels with respect to the pixel of interest. Therefore, the process proceeds to step S4, and the outline rectangle of the pixel of interest itself is registered as a registration rectangle. FIG. 34 (e) shows a state where such registration has been made. As a result, the second registered rectangle is generated.

  Subsequently, the pixel in the third row and the eleventh column becomes the target pixel. Since the target pixel is a foreground pixel and a registration rectangle exists on the left side, a process of expanding the area of the registration rectangle to a circumscribed rectangle including the target pixel is performed. FIG. 34 (f) shows a state where such an extension has been made. The second registration rectangle, which has been only one pixel in size so far, has been expanded to a registration rectangle for two pixels.

  Next, when the pixel in the 4th row and the 2nd column becomes the target pixel, as shown in FIG. 34 (f), the first registered rectangle exists on the upper side, so that the region of the registered rectangle includes the target pixel. Processing to expand to the circumscribed rectangle is performed. FIG. 34 (g) shows a state where such an extension has been made. The first registered rectangle, which has been only 8 pixels in size, has been expanded to a 12-pixel registered rectangle.

  Subsequently, when the pixel in the 4th row, the 3rd column to the 5th column becomes the target pixel, the target pixel itself becomes a pixel in the registration rectangle, and naturally, the registration rectangle exists within a distance of 3 pixels or less. However, since the registration rectangle has already been extended to the circumscribed rectangle including the target pixel, the registration rectangle is not substantially extended. However, when the pixel in the 4th row and the 6th column becomes the target pixel, the first registered rectangle exists on the left side, so that the process of expanding the region of the registered rectangle to the circumscribed rectangle including the target pixel is performed. FIG. 34 (h) shows a state where such an extension has been made. The first registration rectangle, which has been only 12 pixels in size, has been expanded to a registration rectangle of 15 pixels.

  Subsequently, when the foreground pixel in the fourth row and the eleventh column becomes the target pixel, as shown in FIG. 34 (h), the second registration rectangle exists on the upper side. The process of expanding to a circumscribed rectangle is performed. FIG. 35 (i) shows a state where such expansion has been performed. The second registration rectangle that has been only two pixels in size so far has been expanded to a registration rectangle for four pixels.

  When the foreground pixel in the fifth row and the eleventh column becomes the target pixel after the age, as shown in FIG. 34 (i), the second registration rectangle exists on the upper side. The process of expanding to a circumscribed rectangle is performed. FIG. 35 (j) shows a state where such an extension has been made. The second registration rectangle, which has been only a size of 4 pixels so far, has been expanded to a registration rectangle of 6 pixels. Thus, the first registered rectangle that includes the outline of the pixels for 15 pixels and the second registered rectangle that includes the outline of the pixels for 6 pixels are respectively extracted as blob inclusion figures.

  Eventually, in order for the blob inclusion graphic extraction unit 170 to extract the blob inclusion graphic by the above-described method, pay attention to the individual pixels constituting the moving object identification image M in order, and for each target pixel: (a) If the registered rectangle does not exist within a predetermined neighborhood range (distance within 3 pixels in the above example), the pixel forming the outline of the pixel of interest is registered, and (b) If the target pixel is a foreground pixel and there is a registered rectangle in the predetermined neighborhood range, the registration rectangle area is expanded to a circumscribed rectangle including the target pixel, and finally registered. The blob inclusion figure may be extracted based on the registered rectangle.

  Of course, the blob inclusion graphic extraction processing method described above is merely an example, and the blob inclusion graphic extraction processing method performed by the blob inclusion graphic extraction unit 170 illustrated in FIG. 31 is not limited to the above-described method. .

<6-7: Tracker registration unit 160>
The basic function of the tracker registration unit 160 is that the blob inclusion figure B (i) belonging to the i-th time series extracted by the blob inclusion figure extraction unit 170 is used as the i-th time series of the j-th tracker Tj. Is registered in the tracker storage unit 150 as a member belonging to. Actually, the process executed by the tracker registration unit 160 is divided into a subsequent blob inclusion graphic registration process and a new blob inclusion graphic registration process. Here, the subsequent blob inclusion graphic registration process is a first-stage process performed in advance, and the new blob inclusion graphic registration process is a second-stage process performed after the subsequent blob inclusion graphic registration process is executed.

  First, the subsequent blob inclusion figure registration processing is performed for the i-th time with respect to the preceding blob inclusion figure registered in the (i-1) -th time series of the tracker (the blob inclusion figure that is not the last blob inclusion figure). This is a process for registering subsequent blob inclusion figures as time series. Here, the registration of the subsequent blob inclusion graphic is not necessarily performed. If there is no blob inclusion graphic suitable for the subsequent blob inclusion graphic, the registration is not performed and the preceding blob inclusion graphic becomes the last blob inclusion graphic. The continuity of the blob inclusion figure is cut off. The specific method of the subsequent blob inclusion graphic registration process is as described in §4 as the method for recognizing the subsequent blob inclusion graphic.

  That is, this subsequent blob inclusion graphic registration process is performed when a blob inclusion graphic that is not the last blob inclusion graphic is registered in the (i-1) th time series of the jth tracker Tj. As a preceding blob inclusion figure, it is determined whether or not a blob inclusion figure suitable for following is included in the candidate blob inclusion figure for the jth tracker given from the blob inclusion figure extraction unit 170 If it exists, the candidate blob inclusion figure is registered as a subsequent blob inclusion figure belonging to the i-th time series, and if it does not exist, the preceding blob inclusion figure has become the last blob inclusion figure It is comprised by performing the process which shows.

  For example, in the example shown in FIG. 24, since the blob inclusion figure B29a (horse) which is not the last blob inclusion figure is registered in the 29th time series (time t29 column) of the tracker T1, the blob inclusion The subsequent blob inclusion graphic suitable for registering the graphic as the preceding blob inclusion graphic in the 30th time series (time t30 column) is recognized. As described above, the subsequent blob inclusion graphic is selected from, for example, four candidate blob inclusion graphic B30a-1, B30a-2, B30a-3, and B30a-4 as shown in FIG. Of course, if there is no blob inclusion figure suitable as the subsequent blob inclusion figure among these four candidates, the subsequent blob inclusion figure is not registered.

  For this reason, when the tracker registration unit 160 performs the subsequent blob inclusion graphic registration process, for the preceding blob inclusion graphic registered in the (i−1) th time series of the jth tracker Tj, The suitability evaluation of each candidate blob inclusion figure extracted for the jth tracker is performed, and the candidate blob inclusion figure that has obtained the highest evaluation and the evaluation higher than a predetermined standard follows the preceding blob inclusion figure. It is determined that the blob inclusion figure is suitable for the processing, and a process of registering this as a subsequent blob inclusion figure is performed. In the case of the above example, four candidate blob inclusion figures B30a-1, B30a-2, B30a-3, and B30a shown in FIG. 26 (b) with respect to the preceding blob inclusion figure B29a (horse) shown in FIG. 19 (a). If the blob inclusion figure having the highest evaluation is subjected to the evaluation of −4, the registration is registered as the subsequent blob inclusion figure.

  For example, in the case of the example shown in FIG. 26 (b), as described above, the candidate blob inclusion figure B30a-1 obtains the highest evaluation and the evaluation above a predetermined standard. The subsequent blob inclusion figure following the figure B29a (horse) is registered in the time t30 column of the tracker T1. Note that even the candidate with the highest evaluation is not registered as a subsequent blob inclusion graphic if it does not meet the predetermined criteria. For example, in FIG. 26 (b), when the candidate blob inclusion figure B30a-1 does not exist, even if the candidate blob inclusion figure B30a-3 has the highest evaluation, the evaluation satisfies a predetermined standard. Otherwise, the candidate blob inclusion graphic B30a-3 is not registered as a subsequent blob inclusion graphic.

  The suitability evaluation as the subsequent blob inclusion graphic may be performed based on the position, area, and shape similarity between the preceding blob inclusion graphic to be determined and the candidate blob inclusion graphic. FIG. 35 is a plan view showing a method for evaluating the suitability of such subsequent blob inclusion figures.

  FIG. 35 (a) shows a method for evaluating suitability based on positional similarity. Here, the bold rectangle is the preceding blob inclusion figure B (i-1), and the other three rectangles are candidate blob inclusion figures B (i) a, B (i) b, B ( i) c. In the example shown here, the rectangular center points Q, Qa, Qb, and Qc constituting each blob inclusion figure are defined as the positions of each blob inclusion figure, and the smaller the distance between the center points, the higher the similarity is evaluated. A value is given. For example, if the distance between two points Q / Qa, the distance between two points Q / Qb, and the distance between two points Q / Qc are obtained, and the reciprocal of these distances is used as an evaluation value indicating the similarity in position, the similarity An evaluation value quantitatively indicating the sex can be obtained. In the case of the illustrated example, the candidate blob inclusion figure B (i) b located closest to the preceding blob inclusion figure B (i-1) obtains the highest evaluation value.

  Such an evaluation is based on the assumption that if the same moving object moves in a short time, it is likely that the moving object is present at a position closer to the position of another moving object.

  Note that it is not always necessary to use the distance as the evaluation material for positional similarity, and the area of the overlapping portion can be used as the evaluation material. For example, in the case of a moving image captured at a rate of 30 frames / second, the moving time of the moving object in units of frames is only 1/30 seconds, so the position of the moving object on the original image P (i-1) and the original There is no significant difference in the position of the moving object on the image P (i), and the blob inclusion figures that include the outline of the moving object generally overlap each other. And the larger the area of this overlapping portion, the higher the similarity of the position, so that an evaluation value that quantitatively indicates the similarity of the position can be obtained.

  In the case of the example shown in FIG. 35 (a), when the area of the overlapping portion is obtained, B (i-1) and B (i) a do not overlap, so the area becomes 0, and B (i-1) and B (i) b is the area of the hatched portion in the figure, and B (i-1) and B (i) c do not overlap, so the area is zero. Therefore, if the area of the overlapping portion is used as the evaluation value, the candidate blob inclusion figure B (i) b also obtains the highest evaluation value.

  On the other hand, FIG. 35B shows a method for evaluating suitability based on the similarity of areas. That is, let the area of the preceding blob inclusion figure B (i-1) be A, and the area of the candidate blob inclusion figure B (i) a, B (i) b, B (i) c be Aa, Ab, Ac, respectively. For example, an evaluation value indicating the similarity in area can be obtained. For example, if the larger area of two rectangles to be compared is α and the smaller area is β, β / α is an evaluation value that quantitatively indicates the similarity of the areas. The maximum value of the evaluation value β / α is 1, and the closer the evaluation value is to 1, the higher the area similarity.

  Such an evaluation is based on the assumption that the size on the original image will not change suddenly when the same moving object moves in a short time. Of course, when a person walks with their limbs moving, the size on the original image will change slightly. Also, the size will change slightly if the moving object moves closer to or away from the camera. However, if the moving time is about 1/30 second, the size of the moving body will not change so much. Therefore, the suitability evaluation based on the similarity of areas is a rational evaluation method, similar to the suitability evaluation based on the similarity of positions described above.

  FIG. 35 (c) shows a conformity evaluation method based on shape similarity. In the case of the embodiment shown here, since each blob inclusion figure is configured by a rectangle, the shape can be quantified as an aspect ratio (aspect ratio). That is, the shape of the preceding blob inclusion figure B (i-1) can be expressed by the aspect ratio v / h, and the candidate blob inclusion figure B (i) a, B (i) b, B (i) c The shape can also be expressed by aspect ratios va / ha, vb / hb, and vc / hc, respectively. Therefore, for example, if the larger aspect ratio is α and the smaller aspect ratio is β of the two rectangular aspect ratios to be compared, β / α is the shape as in the case of comparing the areas. This is an evaluation value that quantitatively shows the similarity of. The maximum value of the evaluation value β / α is 1, and the closer the evaluation value is to 1, the higher the similarity in shape.

  Such evaluation is based on the assumption that the shape on the original image will not change suddenly when the same moving object moves in a short time. Again, when a person walks while moving their limbs, the shape on the original image will change slightly, but with the same moving object, the shape will not change much. Therefore, the suitability evaluation based on the similarity in shape is a rational evaluation method, similar to the suitability evaluation based on the similarity in position and area described above.

  After all, the suitability evaluation as a subsequent blob inclusion figure compares the preceding blob inclusion figure and the candidate blob inclusion figure to be judged with respect to any one or more combinations of position, area, and shape, and their similarity Based on the above. When similarity evaluation is performed by combining a plurality of evaluation values, each evaluation value obtained quantitatively by the method as described above is multiplied by a predetermined weighting factor, and this is integrated to obtain a comprehensive evaluation value. What should I do?

  If a blob inclusion figure whose evaluation value obtained in this way is equal to or greater than a predetermined reference value does not exist in the candidate blob inclusion figure, there is no blob inclusion figure suitable for registration as a subsequent blob inclusion figure. Become. In this way, in the subsequent blob inclusion graphic registration process, it is suitable for following the preceding blob inclusion graphic B (i-1) registered in the (i-1) th time series of the jth tracker Tj. If the blob inclusion figure does not exist in the candidate blob inclusion figure for the jth tracker, the subsequent blob inclusion figure following the preceding blob inclusion figure B (i-1) is not registered, Processing is performed to indicate that the preceding blob inclusion graphic B (i-1) has become the last blob inclusion graphic.

  Here, as a process indicating that the preceding blob inclusion figure B (i−1) of the jth tracker Tj has become the last blob inclusion figure, the i-th time series of the jth tracker Tj is not included. What is necessary is just to perform the process which makes it a registration state. Here, the unregistered state is a state indicating that there is no blob inclusion figure to be registered, and has a different meaning from an unregistered state where registration has not yet been performed. For example, in the case of the example shown in FIG. 29, the x mark column on the tracker indicates an unregistered state, and the blank column indicates an unregistered state.

  However, in practice, it is not always necessary to distinguish between data indicating an unregistered state and data indicating an unregistered state. For example, although the time t40 column of trackers T4, T5, and T6 shown in FIG. When the graphic registration process) is completed, these blank fields are recognized as x mark fields. Of course, when it is desired to clearly distinguish the unregistered state from the unregistered state, a process of actively writing data indicating the unregistered state (data corresponding to the x mark) may be performed.

  As described above, in the embodiment described so far, the i-th processing is performed to indicate that the preceding blob inclusion graphic B (i−1) of the j-th tracker Tj is the last blob inclusion graphic. The process of making the time series of the unregistered state is performed. For example, the example shown in FIG. 29 is an example in which the moving object O4 (bird) has existed until time t38 as described in §5, but has jumped off the screen at time t39. In this case, even if the process of recognizing the subsequent blob inclusion graphic to be registered in the time t39 field of the tracker T4 does not exist, the candidate blob inclusion graphic suitable for the subsequent blob inclusion graphic does not exist, so the time t39 field is not registered. The process of (x mark) is performed. In other words, the fact that the time t39 field is in the unregistered state (x mark) indirectly indicates that the blob inclusion figure B38d (bird) in the time t38 field is the last blob inclusion figure. Yes.

  In short, in the embodiment described so far, the column of the unregistered state (x mark) functions as a delimiter of a series of blob inclusion graphic groups that are continuous in time series, and a column in which some blob inclusion graphic is registered. If the left column of is in the unregistered state (x mark), the blob inclusion figure is the first blob inclusion figure, and the right column of the column where some blob inclusion figure is registered is unregistered In the case of (x mark), the blob inclusion graphic is a tail blob inclusion graphic. In this way, if the unregistered state (× mark) field is used as a delimiter for a series of blob inclusion figure groups, a series of blob inclusion figure groups sandwiched between the unregistered state (x mark) fields It can be grasped as a tracking result of the same moving object arranged in time series from the blob inclusion figure to the end blob inclusion figure.

  However, the method for indicating the separation of a series of blob inclusion graphic groups is not limited to the method for providing such a column of the unregistered state (x mark). For example, an identification code indicating that the blob inclusion figure is the first blob inclusion figure is attached to the first blob inclusion figure and stored in the corresponding field on the tracker. If an identification code indicating that the figure is a tail blob inclusion figure is attached and stored in the corresponding column on the tracker, the non-registered state (× mark) column is not provided, and the above identification code is used. Thus, it is possible to determine whether each blob inclusion graphic is a head blob inclusion graphic, a tail blob inclusion graphic, or an intermediate blob inclusion graphic sandwiched between them.

  In this way, when the method of discriminating the leading blob inclusion graphic and the trailing blob inclusion graphic by the identification code is adopted, the separation by the column of unregistered state (x mark) becomes unnecessary, so that the trailing blob inclusion graphic is followed. Thus, it is possible to store the leading blob inclusion figure for another moving body (new moving body). For example, in the case of the example shown in FIG. 29, if a new participant appears at time t39, the new blob inclusion figure for the new participant can be registered as the first blob inclusion figure in the time t39 column of tracker T4. In this case, the blob inclusion figure B38d (bird) is registered in the time t38 column of the tracker T4. However, the blob inclusion figure has an identification code that is the last blob inclusion figure, and the time t39 field contains the identification code. Since the registered new blob inclusion figure has an identification code that is the leading blob inclusion figure, there is no problem even if there is no delimiter in the unregistered state (x mark) column.

  In the case of the above example, as a process for indicating that the blob inclusion graphic B38d (bird) in the time t38 field of the tracker T4 has become the last blob inclusion graphic, a process for setting the time t39 field to the unregistered state (x mark). Instead, a process of writing an identification code indicating that the blob inclusion graphic is the last blob inclusion graphic in the blob inclusion graphic B38d (bird) in the time t38 column is performed.

  In the above, the subsequent blob inclusion figure registration process which the tracker registration part 160 performs as a process of the first half step was demonstrated. Next, let's explain the new blob inclusion figure registration process that is performed in the latter half of the process. This new blob inclusion figure registration process is a process of registering a new participant that has newly appeared in the i-th time-series original image P (i) to the tracker as a first blob inclusion figure, and its specific method is As already described in §5 as a method for recognizing a new blob inclusion figure.

  That is, this new blob inclusion figure registration process performs subsequent blob inclusion figure registration processing for each tracker and then adds the subsequent blob inclusion figure registration process to the blob inclusion figure B (i) extracted based on the moving object identification image M (i). When there is a new blob inclusion figure that does not correspond to the subsequent blob inclusion figure registered by the blob inclusion figure registration process, the new blob inclusion figure is used as the first blob inclusion figure for the tracker whose i-th time series is not registered. It is a process to register.

  For example, in the case of the example shown in FIG. 24, the blob inclusion figure B40a (horse), B40b (person), B40c (car) are respectively stored in the time t40 column of the trackers T1, T2, T3 by the subsequent blob inclusion figure registration process in the first half. ) Is registered. Therefore, in the new-stage blob inclusion graphic registration process in the latter half stage, as shown in FIG. 28 (b), four blob inclusion figures B40a, B40b, B40c, and B40e extracted based on the moving object identification image M40 are shown in FIG. It is determined whether there is a new blob inclusion figure that does not correspond to the blob inclusion figure B40a (horse), B40b (person), or B40c (car) registered in the indicated time t40 column. As a result, for the blob inclusion graphic B40e (dog), it is found that the corresponding subsequent blob inclusion graphic is not registered. A process of registering as a leading blob inclusion figure in an arbitrary tracker in the registered state is performed.

  As described in §5, if there are moving objects that overlap each other on the original image, the blob inclusion graphic extracted based on the moving object identification image and the subsequent blob inclusion graphic registration process registered in the first half stage are registered. The correspondence between blob inclusion figures is not one-to-one but one-to-n (n ≧ 2). That is, the correspondence between the rectangles constituting the two blob inclusion figures is not a correspondence relation in which the rectangles constituting the two blob inclusion figures are completely matched, but the correspondence relation including the rectangle constituting the other blob inclusion figure. Even in this case, if the table as shown in FIG. 30 is prepared, the correspondence between the two can be recognized, so the presence or absence of the corresponding subsequent blob inclusion figure can be determined, and the new blob inclusion figure can be determined. Can recognize the existence of

  Eventually, when the tracker registration unit 160 performs the new-stage blob inclusion figure registration process in the second half, the blob inclusion figure B (i) extracted based on the moving object identification image M (i) and the subsequent blob inclusion in the first half stage. Compare the subsequent blob inclusion figures registered in the i-th time series by the figure registration process, and the blob inclusion figures corresponding to each other if they match or one of them includes the other And the blob inclusion figure whose correspondence cannot be recognized in the blob inclusion figure B (i) may be recognized as a new blob inclusion figure.

<6-8: Configuration using computer>
The configuration and operation of the moving object tracking device according to the basic embodiment of the present invention have been described above with reference to the block diagram shown in FIG. 31. In practice, this device incorporates a dedicated program into a computer. Can be configured. For example, the tracker storage unit 150 can be realized by a storage device of a computer, the image input unit 110 can be realized by an input / output interface of a computer, and the other components are such that the computer cooperates with a program. And can be realized as a function to be executed.

<<< §7. Some variations >>>
Subsequently, here, some modified examples of the moving object tracking device according to the basic embodiment described in §6 will be described.

<7-1: Position correction of blob inclusion figure during masking process>
As described in §6, the masking processing unit 140 uses the blob inclusion figure B (i−1) belonging to the (i−1) th time series to identify the i th time series moving object identification image. Since masking is performed for M (i), there is a slight time difference between the mask and the moving object. Due to the existence of such a time difference, for example, in the case of the individual moving object identification image M (30, 1) illustrated in FIG. 26 (a), a part of the moving object eats away from the blob inclusion figures B29b, B29c, B29d used as masks. It has been put out. As a result, a total of four sets of candidate blob inclusion figures B30a-1, B30a-2, B30a-3, and B30a-4 are obtained from the individual moving object identification image M (30, 1) as shown in FIG. Will be extracted.

  Of these four sets of candidate blob inclusion figures, the blob inclusion figure to be followed by the preceding blob inclusion figure B29a (horse) is the candidate blob inclusion figure B30a-1, and the remaining three candidate blob inclusion figures are so-called noise. It is a useless blob inclusion figure that should be called a blob inclusion figure. Of course, even if such noise blob inclusion figure is mixed, the tracker registration unit 160 evaluates the similarity regarding the position, area, and shape, and the candidate most similar to the preceding blob inclusion figure B29a (horse). Since the blob inclusion graphic B30a-1 is selected as the subsequent blob inclusion graphic, no problem occurs. However, the presence of such noise blob inclusion figures is a factor that induces erroneous selection and is preferably avoided as much as possible. That is, ideally, it is preferable to obtain the individual moving body identification image M (30, 1) shown in FIG. 20B instead of the individual moving body identification image M (30, 1) shown in FIG. .

  For this purpose, the position of the blob inclusion figure that functions as a mask may be corrected during the masking process. For example, in FIG. 26 (a), the blob inclusion figure B29b that plays the role of concealing the moving object O2 (person) shows the position of the moving object O2 (person) at the time t29 as shown in FIG. 19 (a). It is shown. However, since the moving object O2 (person) is moving rightward on the screen, as shown in FIG. 26 (a), a blob inclusion figure in which a part of the moving object O2 (person) serves as a mask at time t30. It is sticking out to the right side of B29b. In order to prevent such protrusion, it is presumed that the blob inclusion figure B29b has moved to the right of the screen along with the movement of the moving object O2 (person), and the blob inclusion figure B29b at time t30. And the position correction to the predicted position may be performed on the blob inclusion figure B29b serving as a mask.

  In the example shown in FIG. 26 (a), the blob inclusion figure B29b that functions as a mask for the moving object O2 (person) moving in the right direction is slightly corrected in the right direction, and the moving object O3 moving in the left direction (vehicle) ) And the blob inclusion figures B29c and B29d that function as a mask for the moving object O4 (bird) may be slightly corrected in the left direction. Of course, since such position correction is only correction to the expected position, each moving object may not be completely hidden by the mask after position correction. Since the degree of projection can be reduced, the effect of reducing the area of the noise blob inclusion figure described above can be obtained.

  Thus, in order to correct the position of the blob inclusion figure that functions as a mask at the time of masking processing, the masking processing unit 140 belongs to the (i−1) th consecutive time series stored before each tracker. When a function for obtaining a predicted position of a blob inclusion figure belonging to the i-th time series is added based on the transition of the position of the blob inclusion figure to be generated, and the individual moving object identification image M (i, j) is generated The position correction to the predicted position is performed on the rectangle that becomes the blob inclusion figure B (i-1) belonging to the (i-1) th time series of the non-focused tracker, and the circumscribed position after the position correction Masking for the i-th moving body identification image M (i) may be performed using a rectangle.

  For example, in the example shown in FIG. 26 (a), the predicted position at the time t30 for the blob inclusion figure B29b that functions as a mask for the moving object O2 (person) is the column before the time t29 of the tracker T2 shown in FIG. Can be obtained based on the transition of the position of the blob inclusion figure stored in the. Hereinafter, two examples of specific methods for obtaining the predicted position of the next time point (time t (i)) based on the transition of the past position (position before time t (i-1)). Just to mention.

  FIG. 36 is a plan view showing a first specific method for obtaining such a predicted position. Here, a blob inclusion figure B (shown in the figure) is shown in each column of times t (i-4), t (i-3), t (i-2), and t (i-1) of the same tracker. i-4), B (i-3), B (i-2), and B (i-1) in a registered state, the prediction of the series of blob inclusion figures at time t (i) A specific method for obtaining the blob inclusion figure B (i) ′ indicating the position will be described. Here, the center point of the rectangle constituting each blob inclusion graphic is defined as reference points R0 to R4 of the position of the blob inclusion graphic. Of course, the reference point for the position of each blob inclusion figure is not necessarily defined at the center point of the rectangle, and may be defined at the upper left corner of the rectangle, for example. Alternatively, the barycentric point of the foreground area F may be obtained at the time of extracting the blob inclusion graphic, and this barycentric point may be used as a reference point for the position of the blob inclusion graphic.

  The main point of this first specific method is that from the reference point of the position of the blob inclusion figure B (i-L) belonging to the (i-L) -th (L is an integer of 2 or more) time series. (I-1) Extending the length of the vector toward the reference point at the position of the blob inclusion figure B (i-1) belonging to the 1st time series by (L / (L-1)) times. The tip of the vector obtained by the above is used as the reference point of the predicted position.

  The example shown in FIG. 36 is a case where L = 4, and from the reference point R4 at the position of the blob inclusion figure B (i-4) registered in the time t (i-4) column, the time t (i− 1) The length of the vector “R4 → R1” directed to the reference point R1 at the position of the blob inclusion figure B (i−1) registered in the column is increased by (4 / (4-1)) times. The point R0 at the tip of the vector “R4 → R0” obtained by extending is obtained as the reference point of the predicted position.

  On the other hand, FIG. 37 is a plan view showing a second specific method for obtaining the predicted position. The main point of this second concrete method is that the (i-1) th from the blob inclusion figure B (iL) belonging to the (i-L) th (L is an integer of 3 or more) time series. A quadratic curve C that approximately passes through the reference points at the respective positions of the total L blob inclusion figures up to the blob inclusion figure B (i-1) belonging to the th time series is obtained. The point is to use the point as a reference point of the predicted position. As a specific method for obtaining such a quadratic curve C, methods based on various algorithms are known, and thus detailed description thereof is omitted here.

  The example shown in FIG. 37 is a case where L = 4, and is registered in the time t (i-1) column from the blob inclusion figure B (i-4) registered in the time t (i-4) column. A quadratic curve C that approximately passes through the reference points R4, R3, R2, and R1 at each position of the total four blob inclusion figures up to the blob inclusion figure B (i-1) is obtained. The upper point R0 is obtained as the reference point of the predicted position. In order to obtain the point R0 on the quadratic curve C, for example, the distances along the quadratic curve C between the two points R4 / R3, between R3 / R2, and between R2 / R1 are obtained, and only the average of these distances A point on the quadratic curve C separated from R1 may be a point R0.

  The method illustrated in FIG. 36 is a method of determining the reference point R0 of the predicted position by so-called primary interpolation, and the method illustrated in FIG. 37 is a method of determining the reference point R0 of the predicted position by so-called secondary interpolation. In general, the latter method can be predicted more accurately. In either case, masking is performed after the position of the blob inclusion figure B (i−1) is corrected to the position of the blob inclusion figure B (i) ′. Of course, the blob inclusion figure B (i) ′ merely indicates the position of the mask whose position has been corrected by prediction, and the blob inclusion figure B (i) actually registered in the time t (i) column of the tracker. Are different.

<7-2: Extended correction of blob inclusion figure during masking process>
In §7-1 described above, a method of correcting the position of the blob inclusion figure during the masking process is described in order to reduce the noise blob inclusion figure. However, the method described here is a blob inclusion figure for the same purpose. This is a method of expanding and correcting the size of the.

  When this method is adopted, the masking processing unit 140 generates the individual moving object identification image M (i, j), and the blob inclusion figure B belonging to the (i−1) th time series of the tracker not of interest. The rectangle which becomes (i-1) is corrected to extend the outline outward by a predetermined amount, and the expanded corrected graphic is used to determine the i-th moving object identification image M (i). Masking may be performed.

  FIG. 38 is a plan view showing a specific example of such extended correction. FIG. 38 (a) is a plan view showing the individual moving body identification image M (30, 1), similarly to FIG. 26 (a). Here, the blob inclusion figures B29b, B29c, and B29d are blob inclusion figures that serve as masks, and as described above, a part of the moving object protrudes from these masks. In §7-1 described above, an approach to eliminate this protruding state by correcting the position of the mask is taken, but here, an approach of concealing the protruding portion by expanding the mask is used. Taken. That is, rectangles B29b ′, B29c ′, and B29d ′ indicated by broken lines in the figure are obtained by correcting the rectangles that become the blob inclusion figures B29b, B29c, and B29d by extending the outline outward by a predetermined amount. It is a mask figure after extended correction.

  When the blob inclusion graphic extraction process is performed from the individual moving object identification image M (30, 1) subjected to the masking process using the extended corrected mask graphic as shown in FIG. A set of candidate blob inclusion figures B30a-1, B30a-3, and B30a-4 is extracted. Comparing FIG. 38 (b) with FIG. 26 (b), it can be seen that the candidate blob inclusion figure B30a-2 disappears and the areas of the candidate blob inclusion figures B30a-3 and B30a-4 are reduced. That is, the effect of reducing the noise blob inclusion figure is obtained. Of course, such extended correction and the position correction described in §7-1 may be used in combination.

<7-3: Selection of blob inclusion figure during masking process>
In the present invention, the reason why the individual moving object identification image is created by the masking process by the masking processing unit 140 is to solve the problem when a plurality of moving objects intersect. Therefore, it is not always necessary to conceal a moving object that does not overlap with another moving object by masking.

  For example, in the case of the example shown in FIG. 26 (a), in order to generate the individual moving object identification image M (30, 1) for the target tracker T1 in which the blob inclusion figure for the moving object O1 (horse) is registered, Masking is performed using the inclusion figure B29b (person), the blob inclusion figure B29c (car), and the blob inclusion figure B29d (bird). However, in this example, at time t30, the moving body O1 (horse), the moving body O2 (person), and the moving body O3 (car) overlap, but the moving body O4 (bird) does not overlap. Therefore, for the moving object O4 (bird), there is no problem even if the masking process using the blob inclusion figure B29d (bird) is not performed. That is, in the example shown in FIG. 26 (b), even if the blob inclusion figure corresponding to the whole moving object O4 (bird) remains as the candidate blob inclusion figure instead of the blob inclusion figure B30a-4, the subsequent blob inclusion figure The candidate selected as the graphic is still the blob inclusion graphic B30a-1 (horse).

  Considering such a point, it can be understood that it is not always necessary to use all blob inclusion figures registered in the non-target tracker for the masking process by the masking processing unit 140. For example, when the masking processing unit 140 generates the individual moving object identification image M (i, j), the blob inclusion figure registered in the (i-1) th time series of the non-focused tracker is focused on. Masking may be performed using only the blob inclusion figure located within a predetermined neighborhood range of the blob inclusion figure registered in the (i-1) th time series of the tracker.

  In the case of the example shown in FIG. 26 (a), when generating the individual moving object identification image M (30, 1) for the target tracker T1 in which the blob inclusion figure for the moving object O1 (horse) is registered, the non-target tracker is generated. Among the blob inclusion figures registered in the time t29 field of T2 to T4, the blob inclusion figure B29b located within a predetermined vicinity range of the blob inclusion figure B29a (horse) registered in the time t29 field of the target tracker T1 Masking may be performed using only (person) and B29c (car). In this case, masking by the blob inclusion figure B29d (bird) is not performed.

<7-4: Size correction at subsequent blob inclusion figure registration>
In general, it is considered that the blob inclusion figure extracted based on the same moving object does not change abruptly or extremely. In the transition of the blob inclusion graphic illustrated in FIG. 36 and FIG. 37, each blob inclusion graphic is indicated by a rectangle of the same size based on such a concept.

  However, since animals such as humans, horses, and birds are moving bodies that constantly change their forms, if the extracted blob inclusion figures are arranged in time series, there will be a considerable change in size. Even a moving object such as a car changes its size in the blob inclusion figure if it changes its direction or moves closer to or away from the camera.

  FIG. 39 (a) is a plan view showing an example in which a change in size occurs as a result of arranging blob inclusion figures extracted from the same moving object in time series. Here, it is assumed that the moving path of the moving object is a straight line as shown by a broken line, and the center point of each blob inclusion figure moves from left to right along the broken line. The illustrated blob inclusion figures B1 to B7 are blob inclusion figures extracted at times t1 to t7 based on the same moving object, and are drawn with exaggerated temporal changes in size for convenience of explanation. It is.

  Since the blob inclusion figures B1 to B7 are extracted as circumscribed rectangles of the same moving object, they are correct information as information indicating the area shown on the moving object image. However, when such a sequence of blob inclusion figures B1 to B7 is registered on the tracker, a partial image is cut out from each original image based on these blob inclusion figures, and the moving object is zoomed up. When the moving image to be tracked is presented, there is an adverse effect that the zoom-up magnification varies according to the variation in the size of the blob inclusion figures B1 to B7.

  For example, in the zoom-up image shown in the lower part of FIG. 7, since the size of the blob inclusion graphic B30 and the blob inclusion graphic B40 are equal, the zoom-up magnification is constant. Therefore, even if the moving image presentation is performed such that the moving object (car) is tracked in a zoomed-up state, there is no sense of incongruity. However, as shown in FIG. 39 (a), when tracking is performed by zooming up a moving object based on the blob inclusion figures B1 to B7 whose sizes vary, the size of the moving object changes on the screen. The video must be very difficult to watch.

  In order to solve such a problem, when the blob inclusion figures B1 to B9 shown in FIG. 39 (a) are registered in the tracker as subsequent blob inclusion figures, the size is corrected, for example, FIG. The blob inclusion figures B1 'to B9' as shown in b) may be registered, and the corrected blob inclusion figures may be registered on the tracker.

  Specifically, when the tracker registration unit 160 registers the subsequent blob inclusion graphic for the jth tracker Tj in the subsequent blob inclusion graphic registration process, the size of the subsequent blob inclusion graphic is continuously preceded. What is necessary is just to correct | amend the magnitude | size of the said subsequent blob inclusion figure so that it may become the average size of the predetermined number of registered blob inclusion figures. For example, when the blob inclusion graphic B1 is the first blob inclusion graphic (new blob inclusion graphic), the blob inclusion graphic B1 is registered with the same size, and the subsequent blob inclusion graphic B2 is subsequently registered with the blob inclusion graphic B1, It is registered with the correction to be the average size of B2, and the subsequent subsequent blob inclusion figure B3 is registered with the correction to be the average size of the blob inclusion figures B1 to B3. is there. If a correction is made in advance so as to make the average size of the past n blob inclusion figures, it is possible to follow the gradual fluctuation of the moving object. Note that the blob inclusion figures B1 ′ to B9 ′ shown in FIG. 39 (b) are all the same size and do not represent the blob inclusion figures obtained by the above method.

<7-5: Motion recognition of blob inclusion figure extraction unit>
A specific method of blob inclusion graphic extraction processing by the blob inclusion graphic extraction unit 170 is as described with reference to the flowchart of FIG. 32, and FIG. 34 shows an example of the blob inclusion graphic extraction process by the method. It was. This blob inclusion figure extraction process is a process of recognizing a group of pixels (blob) constituting the foreground region F as a moving object based on the moving object identification image M and extracting a circumscribed rectangle of the pixel group. Actually, the foreground area F on the moving object identification image M is not necessarily an area where a moving object exists.

  As shown in FIG. 5, the moving body identification image M is an image generated by similarity determination between a pixel on the original image P (i) and a pixel on the background image A (i−1). Even if the pixel belongs to the background region K, if it is determined as dissimilar for some reason, the pixel belongs to the foreground region F. For example, when wind acts on the leaves of a tree or the like, the reflected state of light changes, so that there is a possibility that the foreground region F is determined. Further, in the background including the sea, river, pond, etc., the fluctuation of the water surface may be determined as the foreground region F.

  Therefore, in practice, it is preferable that the blob inclusion graphic extraction unit 170 does not recognize a pixel group whose area does not satisfy a predetermined standard as a moving object and does not extract a blob inclusion graphic. In general, an object that is recognized as a moving object on the screen and is to be tracked is considered to have a certain size on the screen, so a pixel group whose area does not meet a predetermined standard is not recognized as a moving object. However, there will be no problem. Then, by not extracting a pixel group having a small area as a blob inclusion figure in this way, the number of candidate blob inclusion figures can be reduced and the calculation burden can be reduced.

<7-6: Variation of figures composing blob inclusion figure>
In the embodiment described so far, the blob inclusion figure extraction unit 170 extracts a circumscribed rectangle of a group of pixels (blob) constituting the foreground region F as a blob inclusion figure. It does not necessarily have to be the bounding rectangle of the blob. For example, any shape of circumscribed figure such as a circumscribed circle or circumscribed triangle may be extracted as a blob inclusion figure. In the embodiments described so far, a rectangle having four sides parallel to the vertical and horizontal directions of the pixel array (so-called regular rectangle) is used as the circumscribed rectangle. Of course, the rectangle extracted as the blob inclusion figure is It does not necessarily have to be a regular rectangle.

  Further, the graphic extracted as the blob inclusion graphic does not necessarily need to be a graphic (circumscribed graphic) in contact with the foreground area F (blob). For example, a figure obtained by enlarging the circumscribed figure may be extracted as a blob inclusion figure. Of course, a graphic showing the fine outline shape of the foreground area F (blob) as it is may be extracted as a blob inclusion graphic. In short, in implementing the present invention, the blob inclusion figure can be constituted by an arbitrary figure as long as it shows a rough outline of a group of pixels (blob) constituting the foreground region F.

  In short, the “blob inclusion figure” referred to in the present application can be a figure having various shapes, such as a figure of any shape circumscribing a group of pixels (blob) constituting the foreground region F, It is possible to use a wide variety of figures of any shape that can be obtained by extending the figure to the outside. However, in practice, it is preferable to use a circumscribed rectangle made of a regular rectangle as the “blob inclusion figure” as in the embodiments described so far. If a circumscribed rectangle consisting of a regular rectangle is used as a blob inclusion figure, as shown in FIG. 13, it can be defined only by the coordinate values of two diagonal vertices, and the calculation burden is to cut out an internal image. Will also be lighter.

<<< §8. Basic technical idea of the present invention >>
The present invention has been described with reference to the basic embodiment and the modifications thereof. Here, the basic technical idea of the present invention will be summarized.

  First, the moving object tracking device according to the present invention is a device having a function of detecting and tracking a moving object in a moving image, and its basic configuration is as shown in the block diagram of FIG.

  That is, the moving object tracking device includes an image input unit 110 that inputs a moving image composed of a plurality of original images P (i) that are continuous in time series, and a background image providing unit that provides a background image that constitutes the background of the moving image. 120, the i th original image P (i) and the background image are compared to generate the i th moving body identification image M (i) for distinguishing the background area from the foreground area. 130, and tracker information constituted by arranging the blob inclusion figure consisting of the inclusion figure of the pixel group estimated to be the same moving object in time series from the first blob inclusion figure to the last blob inclusion figure, A tracker storage unit 150 for storing each moving object, and a moving object other than the moving object of interest included in the (i-1) th original image P (i-1) stored in the tracker storage unit 150 are shown. A masking processing unit 140 that performs masking processing on the foreground region of the moving object identification image M (i) using the lob inclusion figure, and a masked portion of the moving object identification image M (i) before and after the masking process are excluded. A group of pixel groups constituting the foreground region is recognized as a moving object, and a block inclusion figure extraction unit 170 that extracts an inclusion figure of the pixel group as a blob inclusion figure and a tracker storage unit 150 (No. i-1) The preceding blob inclusion figure indicating the target moving object included in the first original image P (i-1) is the same as the target moving object included in the i-th original image P (i). Candidates extracted from the moving object identification image M (i) masked by using the blob inclusion graphic indicating the moving object other than the moving object of interest, the blob inclusion graphic indicating the estimated moving object The lob inclusion figure is selected from the lob inclusion figures and registered in the tracker storage unit 150 as the i th time-series subsequent blob inclusion figure following the preceding blob inclusion figure, and the moving object identification image M (i before masking processing) ) The newly added blob inclusion figure for which the corresponding subsequent blob inclusion figure has not been registered is registered as the first blob inclusion figure in the tracker whose i-th time series is not registered. And a tracker registration unit.

  Here, the tracker registration unit 160 compares the preceding blob inclusion graphic and each candidate blob inclusion graphic with respect to any one or a plurality of combinations of position, area, and shape, and the subsequent blob inclusion graphic based on the similarity. Among the individual blob inclusion figures extracted from the moving object identification image M (i) before the masking process, or the blob inclusion figure registered as the subsequent blob inclusion figure of the i-th time series or A blob inclusion figure that is not in an inclusion relationship is defined as a new blob inclusion figure.

  On the other hand, the present invention can also be understood as a moving object tracking method for detecting a moving object in a moving image and tracking the moving object. FIG. 40 is a flowchart showing the procedure of this moving object tracking method.

  As shown in the figure, this moving body tracking method includes an image input stage S10 in which a computer inputs a moving image composed of a plurality of original images P (i) that are continuous in time series, and a background in which the computer constitutes the background of the moving image. A moving object identification image generation step S20 that compares the image with the i-th original image P (i) to generate an i-th moving object identification image M (i) that distinguishes the background region and the foreground region; In the i-th original image P (i), the computer compares the preceding blob inclusion figure including the outline of the moving object of interest included in the (i-1) -th original image P (i-1). A new blob inclusion that recognizes the subsequent blob inclusion figure that includes the outline of the moving object that is estimated to be the same as the moving object of interest and that includes the outline of the moving object that newly appears in the i-th original image P (i). Recognize shapes Has a junior blob subsumption figure recognition step S30, the.

  Here, the subsequent / new entry blob inclusion figure recognition step S30 uses a blob inclusion figure that includes the outline of a moving object other than the moving object of interest included in the (i-1) th original image P (i-1). Then, a masking process step S31 for performing masking processing on the foreground region of the moving object identification image M (i) and a foreground region excluding the masked portion are configured for the moving object identification image M (i) before and after the masking process. A blob inclusion figure extraction step S32 that recognizes a group of pixels as a moving object and extracts a figure including the outline of the pixel group as a blob inclusion figure, and a blob inclusion figure that includes a moving object other than the target movement. Preceding blob inclusion that includes the contour of the moving object of interest from among the candidate blob inclusion figures extracted from the masked moving object identification image M (i) A subsequent blob inclusion graphic selection step S33 for selecting a blob inclusion graphic suitable for following the shape as a subsequent blob inclusion graphic, and a blob inclusion graphic extracted from the moving object identification image M (i) before the masking process. A new blob inclusion figure recognition step S34 for recognizing a new blob inclusion figure in which a subsequent blob inclusion figure is not recognized as a new blob inclusion figure that includes the outline of a new moving object.

  In the subsequent blob inclusion graphic selection step S33, the preceding blob inclusion graphic and each candidate blob inclusion graphic are compared with respect to any one or a plurality of combinations of position, area, and shape, and the subsequent blob inclusion graphic is selected based on the similarity. The inclusion figure is selected, and in the new blob inclusion figure recognition step S34, the individual blob inclusion figure extracted from the moving object identification image M (i) before the masking process is recognized as the subsequent blob inclusion figure. A process is performed in which a blob inclusion figure that does not match or contain the existing blob inclusion figure is used as a new blob inclusion figure.

<<< §9. Application to monitoring system >>
Finally, an embodiment in which the moving body tracking apparatus according to the present invention is applied to a monitoring system will be described with reference to the block diagram shown in FIG. In this monitoring system, a video camera 210 and a partial moving image presentation unit 220 are added to the moving object tracking device shown in FIG.

  Here, the video camera 210 is a component that is installed at a fixed point, captures a surrounding moving image, and gives it to the image input unit 110. On the other hand, the partial video presenting unit 220 sequentially extracts the blob inclusion figure B (i) continuously registered in one selected tracker from the tracker storage unit 150, and the original image P (i) input by the image input unit 110. ) From the extracted blob inclusion figure B (i) is cut out and displayed to display a partial moving image that tracks a specific moving object.

  For example, when tracker information as shown in FIG. 24 is stored in the tracker storage unit 150, when the user gives an instruction to select the tracker T1 to the partial video presentation unit 220, the partial video presentation unit 220 is displayed. , B40a are sequentially extracted from the head of the tracker T1, and based on these blob inclusion figures, from the original images P1, P2, P3,..., P40, respectively. A process of cutting out and presenting a partial image including each blob inclusion figure is performed. As a result, a moving image in which the moving object O1 (horse) is zoomed in and tracked is displayed on the screen.

  Of course, it is also possible to perform predetermined moving body tracking based on the moving image captured by the video camera 210 in real time. In this case, since the blob inclusion figure of each time column is registered in real time in the tracker storage unit 150, the partial moving picture presentation unit 220 at the time when the blob inclusion figure is registered for the selected tracker, The blob inclusion figure is extracted and the partial image is cut out.

  Since the aspect ratio of the display screen for displaying the image is different from the aspect ratio of the blob inclusion graphic, the partial video presentation unit 220 expands the blob inclusion graphic B (i) outward by a predetermined amount. Then, an image inside the expanded inclusion figure may be cut out and displayed. For example, in the case of the example shown in FIG. 7, the aspect ratio of the rectangles (rectangles indicated by bold lines) constituting the blob inclusion figures B30 and B40 is different from the aspect ratio of the display screen. A rectangle constituting the image is expanded outward by a predetermined amount, and an image inside the expanded rectangle (the contours of the partial images f30 and f40) is cut out. By doing so, it is possible to perform zoom-up display in a state where the moving object to be tracked is always correctly displayed on the screen.

110: Image input unit 120: Background image providing unit 130: Moving object identification image generation unit 140: Masking processing unit 150: Tracker storage unit 160: Tracker registration unit 170: Blob inclusion figure extraction unit 210: Video camera 220: Partial video presentation unit A, Aa, Ab, Ac: Area of the blob inclusion figure A (i-1), A (i): Background image (average image)
a (i-1), a (i): One pixel constituting the background image or its pixel values B, B1-B7, B1'-B7 ': Blob inclusion graphic B30-B40: Blob inclusion graphic B10a-B10d: Blob inclusion graphic B20a-B20d: Blob inclusion graphic B29a-B29d: Blob inclusion graphic B30a-B30g: Blob inclusion graphic B30a-1 to B30a-5: Blob inclusion graphic B40a-B40e: Blob inclusion graphic B (i-4) -B (I-1): Blob inclusion graphic B1 (i-1) to Bj (i-1): Blob inclusion graphic B1 (i) to Bj (i): Blob inclusion graphic B (i) a, B (i) b , B (i) c: Blob inclusion figure B (i) ': Blob inclusion figure predicted position B29b', B29c ', B29d': Extended corrected mask figure C: Quadratic curve F , F20 to F40, F (i): Foreground areas F10a to F10d: Foreground areas F20a to F20d: Foreground areas F30a to F30d: Foreground areas F40a to F40e: Foreground areas f30, f40: Partial images h, ha, hb, hc: Horizontal dimensions K, K10 to K40, K (i) of background inclusion M10, M40, M (i): moving object identification image M (i, j), M30a to M30d: individual moving object identification image m (i) : One pixel constituting the moving object identification image or its pixel values O1 to O5: Moving object P (1) to P (i): Original image (individual still images constituting the moving image)
P10 to P40: Original image (individual still images constituting a moving image)
p (i): one pixel constituting the original image or its pixel values Q1 (x1, y1), Q2 (x2, y2): rectangular vertex coordinates Q, Qa to Qc: reference point R0 of the position of the blob inclusion figure R4: Reference points S1 to S7, S10 to S33 of the position of the blob inclusion figure: steps T1 to T6 and Tj of the flowchart: trackers t10 to t40, t (i-1), t (i): times v, va, vb, vc: vertical dimension of blob inclusion figure w: parameter indicating weight

Claims (25)

A moving body tracking device that detects and tracks a moving object in a moving image,
An image input unit for inputting a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
A background image providing unit for providing a background image constituting a background of the moving image,
A moving object identification image generating unit that compares the i-th original image P (i) with the background image to generate an i-th moving object identification image M (i) that distinguishes a background region from a foreground region;
Based on the i th moving object identification image M (i), a group of pixels constituting the foreground area is recognized as a moving object, and the inclusion figure of the pixel group is included in the i th time series blob inclusion. A blob inclusion figure extraction unit that extracts as figure B (i);
Tracker information configured by arranging multiple blob inclusion figures estimated to be extracted based on the same moving object in time series from the first blob inclusion figure to the end blob inclusion figure is stored for each moving object. A tracker storage unit,
The blob inclusion figure B (i) belonging to the i-th time series extracted by the blob inclusion figure extraction unit is the i-th of the j-th tracker Tj (where j is a natural number indicating the tracker number). As a member belonging to the time series of, the tracker registration unit to register in the tracker storage unit,
A masking processing unit that performs masking using a specific blob inclusion figure stored in the tracker storage unit for the foreground region of the moving object identification image generated by the moving object identification image generation unit;
With
The masking processing unit uses the jth tracker Tj stored in the tracker storage unit as a target tracker, and the blob inclusion figure B (i−) belonging to the (i−1) th time series of the non-target tracker. 1) is used to generate an individual moving object identification image M (i, j) for the tracker Tj of interest by masking the i-th moving object identification image M (i), and the blob To the inclusion figure extraction unit,
The blob inclusion graphic extraction unit masks the individual moving object identification image M (i, j) in addition to the function of extracting the blob inclusion graphic B (i) based on the moving object identification image M (i). A candidate blob for the jth tracker that recognizes a group of pixels constituting the foreground area excluding the portion as a moving object, and that the inclusion figure of the pixel group recognized as a moving object belongs to the i-th time series It has a function to extract as an inclusion figure,
The tracker registration unit
(a) When a blob inclusion figure that is not the last blob inclusion figure is registered in the (i-1) th time series of the jth tracker Tj, this blob inclusion figure is used as the preceding blob inclusion figure. To determine whether there is a blob inclusion figure suitable for following the candidate blob inclusion figure for the jth tracker, and if so, the candidate blob inclusion figure is the i th time series A subsequent blob inclusion figure registration process for performing processing to indicate that the preceding blob inclusion figure has become a tail blob inclusion figure when there is no subsequent blob inclusion figure belonging to
(b) After performing the subsequent blob inclusion graphic registration process for each tracker, the subsequent blob inclusion graphic registration in the blob inclusion graphic B (i) extracted based on the moving object identification image M (i) If there is a new blob inclusion figure that does not correspond to the subsequent blob inclusion figure registered by processing, the new blob inclusion figure is registered as the first blob inclusion figure in the tracker whose i-th time series is not registered. Inclusion pattern registration processing;
A moving body tracking device characterized by The moving body tracking device according to claim 1,
A moving object tracking device, wherein a background image providing unit has a function of capturing and storing a still image given from the outside, and provides the still image as a background image. The moving body tracking device according to claim 1,
The moving object tracking, wherein the background image providing unit has a function of obtaining a simple average or a weighted average of the original images input by the image input unit over a predetermined period in the past, and provides the obtained average image as a background image apparatus. In the moving body tracking device in any one of Claims 1-3,
The moving object identification image generation unit compares the color of the pixel at the specific position on the original image P (i) with the color of the pixel at the specific position on the background image, and the degree of approximation of the colors of both pixels is predetermined. If the condition is satisfied, the pixel at the specific position is set as a pixel in the background area. If the predetermined condition is not satisfied, the pixel at the specific position is set as a pixel in the foreground area. ) Is generated. In the moving body tracking device in any one of Claims 1-4,
A moving object tracking device, wherein the blob inclusion graphic extraction unit does not recognize a pixel group whose area does not satisfy a predetermined standard as a moving object and does not extract a blob inclusion graphic. In the moving body tracking device in any one of Claims 1-5,
A moving object tracking device, wherein a blob inclusion figure extraction unit uses a circumscribed rectangle of a pixel group as its inclusion figure. The moving body tracking device according to claim 6,
The blob inclusion figure extraction unit pays attention to the individual pixels constituting the moving object identification image M (i) in order,
(a) When the target pixel is a foreground pixel and there is no registration rectangle in the predetermined vicinity range, a process of registering a rectangle that forms the outline of the target pixel is performed.
(b) If the pixel of interest is a foreground pixel and there is a rectangle registered in the predetermined neighborhood range, the registration rectangle is expanded to a circumscribed rectangle including the pixel of interest and registered.
A moving object tracking device for extracting a blob inclusion figure based on a finally registered rectangle. The moving body tracking device according to claim 6 or 7,
A tracker storage unit that stores coordinate values of two vertices positioned at diagonals of a circumscribed rectangle as tracker information. In the moving body tracking device in any one of Claims 1-8,
When the masking processing unit generates the individual moving object identification image M (i, j), the blob inclusion figure registered in the (i-1) -th time series of the non-focused tracker is the first of the focused tracker. (I-1) A moving object tracking device characterized in that masking is performed using only a blob inclusion figure located within a predetermined neighborhood range of a blob inclusion figure registered in the first time series. In the moving body tracking device in any one of Claims 1-9,
The blob inclusion belonging to the i-th time series based on the transition of the position of the blob inclusion figure belonging to the (i−1) -th consecutive time series stored in each tracker. When generating the individual moving object identification image M (i, j), the function includes a function for obtaining a predicted position of the figure, and the blob inclusion figure B (i belonging to the (i-1) th time series of the tracker not of interest -1) is subjected to position correction to the predicted position, and masking is performed on the i-th moving object identification image M (i) using the inclusion figure after position correction. Tracking device. The moving body tracking device according to claim 10,
The masking processing unit performs the (i−1) th from the reference point at the position of the blob inclusion figure B (i−L) belonging to the (i−L) th (L is an integer of 2 or more) time series. The tip of the vector obtained by extending the length to the reference point at the position of the blob inclusion figure B (i-1) belonging to the time series is multiplied by (L / (L-1)) times. A moving object tracking device, characterized by using a reference point for a predicted position. The moving body tracking device according to claim 10,
The masking processing unit belongs to the (i-1) th time series from the blob inclusion figure B (iL) belonging to the (i-L) th (L is an integer of 3 or more) time series. A quadratic curve that approximately passes through the reference points at each position of the total L blob inclusion figures up to the blob inclusion figure B (i-1) is obtained, and a point on the quadratic curve is set as a reference point of the predicted position. A moving body tracking device characterized by that. In the moving body tracking device in any one of Claims 1-12,
When the masking processing unit generates the individual moving object identification image M (i, j), the blob inclusion figure B (i-1) belonging to the (i-1) th time series of the tracker not of interest The moving body tracking is characterized in that the contour line is corrected to extend outward by a predetermined amount, and masking is performed on the i-th moving body identification image M (i) by using the inclusion figure after the extended correction. apparatus. In the moving body tracking device in any one of Claims 1-13,
When the tracker registration unit performs the subsequent blob inclusion graphic registration process, the jth track inclusion graphic is registered with respect to the preceding blob inclusion graphic registered in the (i-1) th time series of the jth tracker Tj. The suitability evaluation of individual candidate blob inclusion figures extracted for tracker is performed, and the candidate blob inclusion figure with the highest evaluation and the highest evaluation is followed by the preceding blob inclusion figure. A moving body tracking apparatus characterized by determining a suitable blob inclusion figure and registering it as a subsequent blob inclusion figure. The moving body tracking device according to claim 14,
The tracker registration unit compares the preceding blob inclusion figure to be determined and the candidate blob inclusion figure with respect to any one or a combination of position, area, and shape, and performs suitability evaluation based on the similarity. A moving body tracking device characterized by that. In the moving body tracking device in any one of Claims 1-15,
When the tracker registration unit registers the subsequent blob inclusion graphic for the jth tracker Tj in the subsequent blob inclusion graphic registration process, the size of the subsequent blob inclusion graphic is registered in advance as a predetermined value. A moving body tracking device, wherein the size of the subsequent blob inclusion figure is corrected so as to be an average size of a number of blob inclusion figures. In the moving body tracking device in any one of Claims 1-16,
A blob inclusion figure suitable for the tracker registration unit to follow the preceding blob inclusion figure registered in the (i-1) th time series of the jth tracker Tj in the subsequent blob inclusion figure registration process When the candidate blob inclusion figure for the j-th tracker does not exist in the candidate blob inclusion figure, the preceding blob inclusion figure is obtained by performing the process of making the i-th time series of the j-th tracker Tj unregistered. A moving object tracking device characterized by showing that it becomes an end blob inclusion figure. In the moving body tracking device in any one of Claims 1-17,
When the tracker registration unit performs the new blob inclusion figure registration process, the blob inclusion figure B (i) extracted based on the moving object identification image M (i) and the subsequent blob inclusion figure registration process are the i-th time. The subsequent blob inclusion graphic registered in the series is compared, and if both match or one of them includes the other, the blob inclusion graphic corresponding to each other is recognized, and the blob inclusion graphic B A moving body tracking apparatus characterized in that a blob inclusion figure whose correspondence cannot be recognized in (i) is a new blob inclusion figure. A moving body tracking device that detects and tracks a moving object in a moving image,
An image input unit for inputting a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
A background image providing unit for providing a background image constituting a background of the moving image,
A moving object identification image generating unit that compares the i-th original image P (i) with the background image to generate an i-th moving object identification image M (i) that distinguishes a background region from a foreground region;
Tracker information consisting of blob inclusion figures consisting of inclusion figures of pixel groups estimated to be the same moving object in time series from the beginning blob inclusion figure to the end blob inclusion figure, for each moving object A tracker storage to store;
The moving object identification image M is stored using a blob inclusion figure indicating a moving object other than the moving object of interest included in the (i-1) th original image P (i-1) stored in the tracker storage unit. A masking processing unit that performs a masking process on the foreground region of (i);
For the moving object identification image M (i) before and after the masking process, a block that recognizes a group of pixels constituting the foreground area excluding the masked portion as a moving object and extracts the inclusion figure of the pixel group as a blob inclusion figure An inclusion figure extraction unit;
With respect to the preceding blob inclusion figure indicating the moving object of interest contained in the (i−1) th original image P (i−1) stored in the tracker storage unit, the i th original image P ( i) from a moving object identification image M (i) masked by using a blob inclusion graphic indicating a moving object estimated to be the same as the moving object of interest included in i) using a blob inclusion graphic indicating a moving object other than the moving object of interest. Select from among the extracted candidate blob inclusion figures extracted, and register them in the tracker storage unit as the i th time-series subsequent blob inclusion figures following the preceding blob inclusion figure, and the moving object before masking processing Among the blob inclusion figures extracted from the identification image M (i), a new blob inclusion figure for which no corresponding subsequent blob inclusion figure has been registered is represented by the i-th time series. In the recording state tracker, and the tracker registration unit to be registered as a top blob inclusion figure,
A moving body tracking device comprising: The moving body tracking device according to claim 19,
The tracker registration department
Comparing the preceding blob inclusion graphic and each candidate blob inclusion graphic with respect to any one or more combinations of position, area, and shape, and selecting the subsequent blob inclusion graphic based on the similarity,
Of the individual blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the blob inclusion figure registered as the subsequent blob inclusion figure of the i-th time series matches or is included. A moving object tracking device characterized in that a new blob inclusion figure is used as a new blob inclusion figure.   A program that causes a computer to function as the moving object tracking device according to any one of claims 1 to 20. In addition to the moving body tracking device according to any one of claims 1 to 20,
A video camera that captures a video image and gives it to the image input unit;
The blob inclusion figure B (i) successively registered in one selected tracker is sequentially extracted from the tracker storage unit, and the blob inclusion figure B (i) is obtained from the original image P (i) input by the image input unit. A partial video presenting unit that presents a partial video image tracking a specific moving object by cutting out and displaying the internal image of
A monitoring system comprising: The monitoring system according to claim 22,
A monitoring system characterized in that the partial moving picture presentation unit expands the blob inclusion graphic B (i) outward by a predetermined amount and cuts out and displays an image inside the expanded inclusion graphic. A moving body tracking method for detecting and tracking a moving object in a video image,
An image input stage in which a computer inputs a moving image composed of a plurality of original images P (i) continuous in time series (where i is a natural number indicating the order of time series);
The computer compares the background image constituting the background of the moving image and the i-th original image P (i) to distinguish the background region and the foreground region from the i-th moving object identification image M (i ) To generate a moving object identification image generation step;
In the i-th original image P (i), the computer compares the preceding blob inclusion figure including the outline of the moving object of interest included in the (i-1) -th original image P (i-1). A new blob inclusion that recognizes a subsequent blob inclusion figure that includes the outline of a moving object that is estimated to be the same as the included moving object, and that includes a new appearance of the moving object in the i-th original image P (i). Subsequent / newcomer blob inclusion figure recognition stage to recognize the figure,
Have
The subsequent / newcomer blob inclusion figure recognition stage includes:
A foreground region of the moving object identification image M (i) is obtained using a lob inclusion figure that includes the outline of a moving object other than the moving object of interest included in the (i-1) th original image P (i-1). Masking processing step for performing masking processing on,
In the moving object identification image M (i) before and after the masking process, a group of pixels constituting the foreground area excluding the masked portion is recognized as a moving object, and a figure including the outline of the pixel group is defined as a blob inclusion figure. Blob inclusion figure extraction step to extract;
Preceding blob inclusion that includes the outline of the moving object from candidate blob inclusion figures extracted from the moving object identification image M (i) masked using the blob inclusion figure that includes moving objects other than the moving object. A subsequent blob inclusion shape selection step for selecting a blob inclusion shape suitable for following the shape as a subsequent blob inclusion shape;
Among the blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the new blob inclusion figure in which the new blob inclusion figure in which the corresponding subsequent blob inclusion figure is not recognized is included in the outline of the new moving object is included. A new blob inclusion figure recognition step that recognizes as
A moving body tracking method characterized by comprising: The moving body tracking method according to claim 24,
In the subsequent blob inclusion graphic selection step, the previous blob inclusion graphic and each candidate blob inclusion graphic are compared with respect to one or more combinations of position, area, and shape, and the subsequent blob inclusion graphic is determined based on the similarity. Make a selection
In the new blob inclusion figure recognition step, among the individual blob inclusion figures extracted from the moving object identification image M (i) before the masking process, the blob inclusion figure recognized as the subsequent blob inclusion figure is matched or included in the relation A moving object tracking method characterized in that a blob inclusion figure which is not formed is used as a new blob inclusion figure.

Images