JP7272024B2 - Object tracking device, monitoring system and object tracking method - Google Patents

Description

The present invention relates to technology for tracking an object in moving images.

Object tracking, which tracks an object detected in a frame of moving images (time-series images), is an important technique in the field of computer vision.

A method using a correlation filter is known as one of object tracking algorithms (Non-Patent Document 1). In this method, an image feature of a shape such as an HOG feature is used, and online learning is performed including the peripheral area of the tracked object. However, since the above-described feature values are used, tracking may fail when the shape or color of the object changes abruptly. In addition, since online learning is performed using images that include the surrounding area of the tracked object, it sometimes drifts into the background without obtaining an appropriate response under a complex background.

It is also known to use background subtraction for object tracking (eg, Patent Document 1). In the background subtraction method, a difference between frames is obtained, and an area with a large difference value is detected as an object. With this method, if the movement of the tracked object stops, it will be lost. For example, if the object to be tracked is a person, it will be lost if the person sits on a chair, so it is not suitable for monitoring in the office. Furthermore, in template matching, if the object is deformed and the difference from the template exceeds a predetermined threshold, the object is lost. In the case of a person, tracking fails because the motion of the person causes a large deformation compared to the template.

By the way, in the field of building automation (BA) and factory automation (FA), image sensors are used to automatically measure the number, position, and flow of people, and optimally control equipment such as lighting and air conditioning. What is needed is an application that For such applications, an ultra-wide-angle camera equipped with a fish-eye lens (also called a fish-eye camera, omnidirectional camera, omnidirectional camera, etc.) is used to acquire image information in as wide a range as possible. In this specification, the term "fish-eye camera" is used) is often used. Furthermore, in the above applications, in order to acquire image information in as wide a range as possible, a camera mounted on a high place such as a ceiling is arranged so that the viewpoint of the camera is a top view. With a camera of this arrangement, the viewpoint for photographing a person is a front view when the person is in the periphery of the image, and a top view when the person is in the center of the image.

An image captured by a fish-eye camera is deformed due to distortion in the appearance of an object to be captured depending on the position within the imaging plane. Furthermore, when the camera viewpoint is set to the top view, the appearance changes depending on the position of the tracked object. Also, in an environment with limited processing power, such as an embedded device, the frame rate is likely to be low, and there is a peculiarity in that the amount of movement of an object and the amount of feature change between frames are large. Therefore, the tracking method of the prior art may not be able to track accurately.

JP-A-2002-157599

Henriques, Joao F., et al. "High-speed tracking with kernelized correlation filters." IEEE transactions on pattern analysis and machine intelligence 37.3 (2015): 583-596.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an object tracking technique that is more accurate than conventional techniques.

In order to achieve the above objects, the present invention employs the following configurations.

A first aspect of the present invention includes acquisition means for acquiring the position of an object in a first frame image, and tracking for obtaining the position of the object from a second frame image that is a frame image after the first frame image. and means for tracking an object. The tracking means includes a first sub-tracking means, a second sub-tracking means and a locating means. A first sub-tracking means obtains first coordinates of the object in the second frame image by a first tracking algorithm based on the feature amount extracted from the second frame image. The second sub-tracking means obtains second coordinates, which are the center of gravity of the moving area, based on the difference image between the frames of the first frame image and the second frame image. The position specifying means obtains the position of the object in the second frame image based on the first coordinates and the second coordinates.

The object to be tracked, or "object", can be any object, such as a human body, a face, an animal, a vehicle, and the like. The first tracking algorithm can be any object tracking algorithm, but a tracking algorithm with local optimization is preferred. A tracking algorithm based on local optimization is an algorithm that learns and tracks an image of a partial region containing a tracked object. This learning is preferably performed by a sequential learning type tracking algorithm.

The position of the object (first coordinates) identified by the first tracking algorithm may not be accurate due to the effects of error accumulation during learning. Therefore, as described above, the position of the center of gravity (second coordinates) of the moving region is obtained based on the difference image between frames, and the position of the object is specified in consideration of this position of the center of gravity. can be obtained with higher accuracy, and the tracking accuracy is improved.

The second sub-tracking means of the present invention uses a color histogram of an area containing the object generated using at least the first frame image to express the probability that the object exists in the second frame image. generating a likelihood map (heat map), multiplying an inter-frame difference image between the first frame image and the second frame image by the likelihood map to generate an adjusted difference image, and performing the adjustment; In the completed difference image, the center of gravity of the pixel area corresponding to the degree of difference may be obtained as the second coordinates. In this way, by considering not only the difference between frames but also the foreground likelihood based on the color histogram, the tracking accuracy is further improved. Determining the position of the center of gravity based only on the difference between frames is affected by moving objects that are not the target of tracking, but this effect can be eliminated or reduced by considering the foreground likelihood.

Also, the second sub-tracking means of the present invention sets a central region centered on a position where the object is estimated to exist and a peripheral region around the central region in the adjusted difference image. Alternatively, the center of gravity may be obtained by assigning different weights to the pixels in the central area and the pixels in the peripheral area. The weight may be a fixed value, or a value corresponding to the areas of the central region and the peripheral region may be adopted.

Further, the second sub-tracking means of the present invention determines a weighting factor based on the sum of differences in the central area and the sum of differences in the peripheral area, and calculates the first coordinates using the weighting factors. A coordinate determined as a weighted average of the second coordinates may be determined as the position of the object. For example, the larger the sum of the differences in the central region, or the larger the weight for the second coordinate may be set. In addition to this, the weighting factor is set larger as the ratio of the sum of the differences in the central region to the sum of the differences in the peripheral region is larger, or as the sum of the sum of the differences in the central region and the sum of the differences in the peripheral region is larger. You may set it large. The weight for the second coordinate can be regarded as the strength of the correction effect when correcting the first coordinate using the second coordinate, or the application rate.

Note that the center region described above can be determined based on the position of the object in the first frame image and the moving speed of the object in the first frame image.

Further, in the present invention, when the position of the object determined by the first sub-tracking means is sufficiently reliable, the effect (application rate) of correction by the second coordinates is set to be smaller than otherwise. Alternatively, no modification by the second coordinates may be performed. That is, when the likelihood (reliability) of the first coordinate obtained by the first sub-tracking means is equal to or greater than the second threshold, the tracking means assigns a smaller weight to the second coordinate than otherwise. to determine the position of the object as a weighted average of the first coordinate and the second coordinate.

In addition, in the present invention, when the positions of the object determined by the first sub-tracking means and the second sub-tracking means are sufficiently close, the effect (application rate) of correction by the second coordinates is made smaller than otherwise. It may be set or may not be modified by the second coordinate. That is, when the difference between the first coordinate and the second coordinate is less than the third threshold, the position identifying means assigns a smaller weight to the second coordinate than otherwise, and and the second coordinate may be used to determine the position of the object.

The first tracking algorithm adopted by the first sub-tracking means in the present invention is not particularly limited. It may be an algorithm for determining the position of an object. Also, the first tracking algorithm may be a sequential learning algorithm that learns each time the position of the object is specified.

Also, the image to be processed in the present invention may be a fish-eye image obtained by a fish-eye camera. A “fish-eye camera” is a camera equipped with a fish-eye lens, and is capable of shooting at a super wide angle compared to a normal camera. An omnidirectional camera, an omnidirectional camera, and a fisheye camera are all types of ultra-wide-angle cameras, and all have the same meaning. The fisheye camera may be installed so as to look down on the detection target area from above the detection target area. The optical axis of the fish-eye camera is typically set vertically downward, but the optical axis of the fish-eye camera may be tilted with respect to the vertical direction. Since the fisheye image has a large distortion, especially in the low frame rate image, the characteristics of the object change greatly between frames, and the drift to the background frequently occurs. Furthermore, when the camera is installed so that the optical axis faces vertically downward, the viewpoint from which the object is captured changes depending on the position of the object in the image. occur frequently. However, according to the present invention, even with such a fisheye image, accurate tracking is possible even if the optical axis of the camera is set vertically downward. However, the image to be processed by the present invention is not limited to the fisheye image, and may be a normal image (an image with little distortion or an image with a high frame rate).

A second aspect of the present invention includes an acquisition step of acquiring a position of an object in a first frame image, and tracking for obtaining the position of the object from a second frame image that is a frame image after the first frame image. and a second frame image of the object by a first tracking algorithm based on the feature amount extracted from the second frame image. determining second coordinates, which are the center of gravity of a motion area, based on the difference image between the frames of the first frame image and the second frame image; and determining the position of the object in the second frame image based on the coordinates and the second coordinates.

The present invention may be regarded as an object tracking device having at least part of the above means, or as an image processing device or a monitoring system. Further, the present invention may be regarded as an object tracking method, an image processing method, and a monitoring method including at least part of the above processing. Further, the present invention can also be regarded as a program for realizing such a method and a recording medium on which the program is non-temporarily recorded. It should be noted that each of the means and processes described above can be combined with each other as much as possible to constitute the present invention.

According to the present invention, object tracking can be performed with higher accuracy than in the past.

FIG. 1 is a diagram showing an application example of a person tracking device according to the present invention. FIG. 2 is a diagram showing the configuration of a monitoring system that includes a person tracking device. FIG. 3 is a detailed functional block diagram of the tracking unit. FIG. 4 is a flowchart of overall processing performed by the person tracking device. FIG. 5 is a flowchart of learning processing. FIG. 6 is a flowchart of tracking processing. FIG. 7 is a flowchart of target center correction processing in tracking processing. FIG. 8 is a flowchart of masking processing for other tracking targets in the target center correction processing. FIG. 9 is a diagram for explaining learning processing and tracking processing (including correction processing) in this embodiment. FIG. 10 is a diagram for explaining the adjusted difference image, the centroid of the difference area, and the sum of the differences between the central area and the peripheral area. FIG. 11 is a diagram illustrating a function used to obtain the application rate. 12A and 12B are diagrams for explaining the masking process for other tracking targets.

<Application example>
An application example of an object tracking device according to the present invention will be described with reference to FIG. The human tracking device 1 analyzes a fisheye image obtained by a fisheye camera 10 installed above a tracked area 11 (for example, a ceiling 12), and detects a person 13 existing within the tracked area 11. • It is a tracking device. This person tracking device 1 detects, recognizes, and tracks a person 13 passing through a tracking target area 11, for example, in an office, a factory, or the like. In the example of FIG. 1, the four human body regions detected from the fisheye image are indicated by bounding boxes. The detection result of the human tracking device 1 is output to an external device and used, for example, for counting the number of people, controlling various devices such as lighting and air conditioning, monitoring suspicious persons, and analyzing flow lines.

In this application example, a tracking algorithm based on local optimization is adopted as the object tracking algorithm. In this algorithm, tracking is performed by learning images of subregions containing the tracked object and locating regions with similar features to the object. Since the vicinity of the object is also included in the learning target, the position of the object cannot be predicted appropriately in situations where the background changes in a complicated manner, resulting in a malfunction called drift, in which the background is mistakenly determined to be the object. I have something to do. In order to prevent such malfunction, in this application example, the object position obtained by the tracking algorithm by local optimization is corrected by the object position (difference centroid) obtained based on the difference image between frames. to suppress drift to the background. More specifically, in addition to the difference image between the previous frame image and the current frame image, considering the color histogram, the moving speed (direction and speed) of the tracked object, the distribution of the difference between the central area and the peripheral area, etc., Calculation of the center of gravity difference and the degree of position correction effect based on the center of gravity difference are determined. This improves basic tracking performance, suppresses background fixation, and achieves more accurate tracking.

<Monitoring system>
An embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of a monitoring system to which the person tracking device according to the embodiment of the invention is applied. A monitoring system 2 includes a fisheye camera 10 and a human tracking device 1. - 特許庁

The fisheye camera 10 is an imaging device having an optical system including a fisheye lens and an imaging element (image sensor such as CCD or CMOS). For example, as shown in FIG. 1, the fisheye camera 10 is installed on the ceiling 12 of the tracking target area 11 or the like with the optical axis directed vertically downward, and captures an omnidirectional (360 degrees) image of the tracking target area 11. Take a picture. The fisheye camera 10 is connected to the person tracking device 1 by wire (USB cable, LAN cable, etc.) or wirelessly (WiFi, etc.). The image data may be either a monochrome image or a color image, and the resolution, frame rate and format of the image data are arbitrary. In this embodiment, it is assumed that color (RGB) images captured at 10 fps (10 images per second) are used.

The human tracking device 1 of this embodiment has an image input unit 20 , a human body detection unit 21 , a learning unit 22 , a storage unit 23 , a tracking unit 24 and an output unit 28 .

The image input unit 20 has a function of capturing image data from the fisheye camera 10 . The captured image data is handed over to the human body detection unit 21 and the tracking unit 24 . This image data may be stored in the storage unit 23 .

The human body detection unit 21 has a function of detecting a human body from a fisheye image using an algorithm for detecting a human body. The human body detected by the human body detection unit 21 is targeted for tracking processing by the tracking unit 24 . Note that the human body detection unit 21 may detect only a person newly appearing in the image, and may exclude the vicinity of the position where the person to be tracked exists from the target of detection processing. Furthermore, a Tracking-by-detection method may be used in which the human body detection unit 21 detects a person in the entire image at a constant time interval or frame interval, and then the tracking unit 24 performs tracking processing.

The learning unit 22 learns the characteristics of the human body to be tracked from the human body image detected by the human body detection unit 21 or specified by the tracking unit 24 and stores the learning result in the storage unit 23 . Here, the learning unit 22 obtains a correlation filter for performing evaluation based on shape features and a color histogram for performing evaluation based on color features. The learning unit 22 performs learning for each frame, and updates the past learning result by reflecting the learning result obtained from the current frame with a predetermined coefficient.

The storage unit 23 stores learning results learned by the learning unit 22 . The storage unit 23 also stores hyperparameters for learning processing and tracking processing, such as feature quantities to be used, parameters for each feature quantity, learning coefficients, and weighting coefficients (functions) for correction using the center of gravity of the difference.

The tracking unit 24 identifies the position of the person to be tracked in the current frame image. The tracking unit 24 first identifies an object position having characteristics similar to those of a person detected from within the target area, with the area including the detection position by the human body detection unit 21 as the target area. After that, the position of the person to be tracked is specified in the current frame image with the vicinity of the position specified by the tracking unit 24 in the previous frame image as the target area.

FIG. 3 is a more detailed functional block diagram of the tracking unit 24. As shown in FIG. The tracking section 24 includes a first sub-tracking section 25 , a second sub-tracking section 26 and a position correction section 27 . The first sub-tracking unit 25 includes a feature extraction unit 251 , a response map generation unit 252 and a provisional center determination unit 253 . The second sub-tracking section 26 includes a difference extraction section 261 , a heat map generation section 262 , a difference image adjustment section 263 , a different target region mask section 264 , a center prediction section 265 , a center of gravity determination section 266 and a weight determination section 267 .

The feature quantity extraction unit 251 extracts a feature quantity relating to the shape and a feature quantity relating to the color of the object from the target region. The feature amount extraction unit 251 extracts the HOG feature amount as the feature amount regarding the shape, and extracts the color histogram as the feature amount regarding the color.

The response map generating unit 26 uses the extracted feature amount and the correlation filter stored in the storage unit 23 to generate a response map (likelihood map).

A temporary center determination unit 253 determines the temporary position of the tracked object in the current frame image based on the response map. Specifically, the provisional center determination unit 253 identifies the position in the target region that takes the maximum value in the response map, and provisionally determines this position as the center position (first coordinate) of the tracked object in the current frame image. do. Note that it is not necessary to determine the position of the maximum value in the response map as the center position. If multiple peaks appear in the response map, the position of the local peak other than the maximum value can be determined by considering other factors. may be tentatively determined as the center position of .

The second sub-tracking unit 26 determines the center position of the tracked object in the current frame image based on the difference between the previous frame image and the current frame image. Each functional unit of the second sub-tracking unit 26 will be described in detail below with reference to flowcharts.

The position correction unit 27 corrects the position of the target object determined by the first sub-tracking unit 25 using the position of the target object determined by the second sub-tracking unit 26 to obtain the center of the target object in the current frame image. Locate. The position correction unit 27 is an example of position specifying means for specifying the position of the object in the current frame image. Specifically, the final center position of the object is identified by taking the weighted sum (weighted average) of the two positions. Details of the processing will be described below with reference to flowcharts.

The output unit 28 has a function of outputting information such as fisheye images, detection results, and tracking results to an external device. For example, the output unit 28 may display information on a display as an external device, may transfer information to a computer as an external device, or may transmit information to a lighting device, an air conditioner, or an FA device as an external device. Information and control signals may be transmitted.

The person tracking device 1 can be configured by, for example, a computer including a CPU (processor), memory, storage, and the like. In that case, the configurations shown in FIGS. 2 and 3 are realized by loading a program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, a smart phone, or a built-in computer such as an on-board computer. Alternatively, all or part of the configuration shown in FIG. 2 may be configured with ASIC, FPGA, or the like. Alternatively, all or part of the configuration shown in FIG. 2 may be realized by cloud computing or distributed computing.

<Overall processing>
FIG. 4 is an overall flowchart of human tracking processing by the monitoring system 2 . The overall flow of human tracking processing will be described along FIG.

First, in step S<b>101 , the user sets learning and tracking hyperparameters for the human tracking device 1 . Examples of hyperparameters include feature quantities to be used, parameters for each feature quantity, learning coefficients, and weighting coefficients (functions) for correction using the center of gravity of the difference. The inputted hyperparameters are stored in the storage unit 23 .

Next, in step S102, the human tracking device 1 acquires a target area. The target area is an area including the area where the person to be tracked exists and its periphery, and is an area where the person to be tracked is likely to exist. The target area can also be said to be an area to be processed by the tracking unit 24 . In this embodiment, the initial position of the person to be tracked is detected by the human body detection unit 21 . However, the initial position of the tracked person may, for example, be entered by the user.

Thereafter, the processing from steps S104 to S107 is repeatedly performed. If the termination condition is satisfied in the determination of termination in step S103, the process is terminated. The termination condition can be, for example, the loss of the tracked person (frame out) or the end of the animation.

In step S<b>104 , the image input unit 20 inputs a one-frame fisheye image from the fisheye camera 10 . At this time, a flat unfolded image in which the distortion of the fisheye image is corrected may be created and subsequent processing may be performed. used for the processing of

In step S105, it is determined whether the current frame is the first image. Here, the first image is a frame image to which the initial position of the person to be tracked is given, typically a frame image from which the person to be tracked has been detected by the human body detection unit 21 .

If the current frame is the image of the frame after the first image, the process proceeds to step S106, and the tracking unit 24 executes the tracking process. Details of the tracking process will be described later.

In step S107, the learning unit 22 executes learning processing based on the area where the target person exists in the current frame image. Details of the learning process will be described later.

In this manner, the position of the person to be tracked is specified for each frame by the tracking process S106, and tracking is realized. Further, the tracking method of this embodiment employs a sequential learning type tracking algorithm that learns the characteristics of the person to be tracked for each frame.

<Learning processing>
FIG. 5 is a flowchart showing details of the learning process in step S107. FIG. 9 is a diagram for explaining the learning process and the tracking process using the learning result. The learning process will be described below with reference to FIGS. 5 and 9. FIG.

The learning unit 22 first cuts out the target area 904 from the current frame image (S201). Here, it is assumed that the T frame is the current frame image. As shown in FIG. 9, target area 904 is an area that includes foreground area 902 and background area 903 of a person. The foreground area 902 is the area where the tracked person exists, and the background area is the area where the tracked person does not exist. The size of the background area 903 is determined according to the size of the foreground area 902 . For example, the size of the background area 903 is determined such that the size of the foreground area 902 is a predetermined ratio (eg, ⅓) of the overall size of the target area 904 . Since the target area is updated at the end of the tracking process so that the center of the target area is the position of the person to be tracked (step S306 in FIG. 6), the center 901 of the target area 904 is equal to the center position of the person to be tracked.

The learning unit 22 acquires the brightness feature amount image 906 and the HOG feature amount image 907 as feature amount images of the target region 904 (S202). The HOG feature amount is a feature amount obtained by forming a histogram of the luminance gradient direction of a local region, and can be regarded as a feature amount representing the shape/contour of an object. Here, the HOG feature amount is used, but other feature amounts representing the shape/contour of the object, such as the LBP feature amount, the SHIFT feature amount, and the SURF feature amount, may be used. Also, a luminance image may be employed instead of the brightness image. Note that if the brightness feature amount image and the HOG feature amount image have been obtained in the tracking process (step S302 in FIG. 6), there is no need to obtain them again.

The learning unit 22 stores the lightness feature amount image 906 in the storage unit 23 so that it can be used when obtaining a difference image between frames in the next frame (S203).

The learning unit 22 also acquires the color histogram 908 within the target region 904 (S204). Specifically, the color histograms of the foreground area 902 and the background area 903 are obtained. A color histogram is a feature quantity representing color, and a Color Names (CN) feature quantity can be adopted as another feature quantity representing color. Also, a luminance histogram may be employed as a feature quantity representing a luminance feature instead of a color feature quantity. The learning unit 22 stores the obtained color histogram 908 in the storage unit 23 .

The learning unit 22 obtains a correlation filter 909 whose response has a peak at the center of the target (S205). Specifically, after extracting the HOG feature amount image 907, the correlation filter 909 is obtained by obtaining a filter that is closest to an ideal response having a peak only at the center with respect to the correlation of the feature amount image itself. can get. When calculating the correlation filter in Fourier space, the feature amount image may be multiplied by a window function. The HOG feature amount image 907 is stored in the storage unit 23 because it is used when applying a correlation filter in tracking processing of the next frame.

If this learning is the first learning (S206-YES), the correlation filter and color histogram generated in steps S204 and S205 are stored in the storage unit 23 as they are. On the other hand, if the current learning is the second or later learning (S206-NO), the process proceeds to step S207.

In step S207, the learning unit 22 obtains a new correlation filter by synthesizing the correlation filter obtained last time (the correlation filter stored in the storage unit 23) and the correlation filter obtained this time in step S205. memorize to Further, in step S208, the learning unit 22 obtains a new color histogram by synthesizing the color histogram obtained last time (the color histogram stored in the storage unit 23) and the color histogram obtained this time in step S204. Stored in the storage unit 23 . Weights (learning coefficients) for synthesis may be determined as appropriate.

<Tracking process>
FIG. 6 is a flowchart showing details of the tracking process in step S106. FIG. 9 is a diagram for explaining the learning process and the tracking process using the learning result. The tracking process will be described below with reference to FIGS. 6 and 9. FIG.

[Tracking processing by correlation filter model]
The tracking unit 24 cuts out the target area 905 from the current frame image (S301). Here, it is assumed that the (T+1)th frame is the current frame image. Note that the target area has been updated at the end of the previous tracking process so that the center of the target person is the position of the person to be tracked (step S306 in FIG. 6). Equal to the center position of the target person. Here, as shown in FIG. 9, a target area 905 corresponding to the target area 904 whose center is the position of the person to be tracked identified in the T-th frame is cut out.

The feature quantity extraction unit 251 extracts the brightness feature quantity image 910 and the HOG feature quantity image 911 as the feature quantity images of the target region 905 (S302). The brightness feature amount image 910 has the same resolution as the frame image, but the HOG feature amount image has a lower resolution than the frame image because the feature amount is obtained for each cell (for example, every 3×3 pixels). The response map generation unit 252 applies a correlation filter 909 to the correlation between the HOG feature amount image 911 in the target region 905 and the HOG feature amount image 907 stored in the storage unit 23 to generate a response map 921 (likelihood map ) is obtained (S303). The tentative center determining unit 253 tentatively determines that the position of the maximum value in the response map 921 is the center position p (922) of the object in the current frame image (S304). Note that the center position p does not need to be determined as the position that takes the maximum value in the response map 921. For example, when a plurality of peaks appear in the response map 921, the positions of the peaks other than the maximum value are determined in consideration of other factors. may be the center position p.

[Correction process]
In step S305, the tracking unit 24 obtains the center position c of the object based on the difference image between the frames, and corrects the center position p obtained in step S304 with the center position c. FIG. 7 is a flowchart showing the details of the processing in step S305. A detailed description will be given below with reference to FIG.

The heat map generation unit 262 calculates the likelihood ( A foreground likelihood map 912 is generated (S401). Foreground likelihood maps based on color histograms are referred to herein as heatmaps.

The tracking unit 24 reduces the resolutions of the heat map 912, the brightness feature amount image 906 of the previous frame, and the brightness feature amount image 910 of the current frame to the same resolution. The tracking unit 24, for example, makes the resolution of these images the same as the response map 921 obtained as a result of the correlation filter calculation. By reducing the resolution, it is possible to obtain an image composed of pixels obtained by averaging the foreground likelihoods and differences in an area having a certain degree of spread.

The difference image adjustment unit 263 calculates the absolute value of the difference (absolute value difference) for each pixel of the low-resolution lightness feature amount images 906 and 910, and generates a difference absolute value image 913 (S404). . The difference image adjustment unit 263 integrates the difference absolute value image 913 and the heat map 912 representing the foreground likelihood for each pixel to generate an adjusted difference image 914 (S404). The adjusted difference image can be regarded as an image obtained by lowering the resolution of a simple difference image between frames and adjusting it according to the foreground likelihood based on color. By reducing the resolution, it is possible to grasp the difference for an area with a certain degree of spread, and by considering the foreground likelihood based on color, it is possible to exclude moving objects that are not the target of tracking.

The difference image adjustment unit 263 further binarizes the adjusted difference image 914 (S405). Any existing dynamic binarization algorithm can be adopted for the binarization process, but binarization may be performed with a fixed threshold. FIG. 10A shows an example of a binarized adjusted difference image 914 . A pixel 1001 shown in white in the figure is a pixel with a difference (a pixel whose difference is above the threshold). A region consisting of all pixels 1001 with a difference corresponds to a moving region.

In the adjusted difference image 914 thus obtained, the other target region masking unit 264 sets a mask region for ignoring (removing) the difference caused by another tracked object (S406). . Here, for ease of understanding, it is assumed that there is no other tracked object within the target area 905, and the description continues without going into details of the mask processing. Details of the mask processing will be described later.

The center prediction unit 265 predicts the center position of the tracked object in the current frame image from the center position of the tracked object in the previous frame image and the movement speed of the tracked object up to the previous frame image (S407). FIG. 10B shows the center position 1002 in the previous frame image and the moving speed 1003 of the tracked object up to the previous frame image. Predict 1004. A Bayesian filter may be used for prediction. For example, the movement position may be estimated using a Kalman filter, or the movement position may be estimated using optical flow. A center area 1005 centered on the predicted center position 1004 is set. Also, an area of the target area other than the central area 1005 is set as a peripheral area.

Next, the center-of-gravity determining unit 266 obtains the position of the center of gravity c of the region composed of the pixels with the difference (S408, reference numeral 915 in FIG. 9). A weight may be set for each pixel when obtaining the center of gravity c. That is, the center of gravity c may be obtained by the following formula (1).
Center of gravity of difference=Σ _i (d(i)×w(i)×p(i)) (1)
Here, p(i) is the pixel coordinate (position vector), d(i) is the presence or absence of a difference (1 if there is a difference, 0 if not), w(i) is the weight, and the sum (sigma) is The entire adjusted difference image is targeted.

For example, the weight w(i) may be set to different values for the pixels in the central region and the pixels in the peripheral region. It is conceivable to set inversely proportional values. Other than this, the weight may be set based on other criteria, such as setting a greater weight as the position is closer to the predicted center position 1004 . A simple center of gravity may be obtained without weighting when obtaining the center of gravity c. Note that the coordinates of the barycentric position may be obtained with sub-pixel precision. Also, the above d(i) has a binary value of 0 or 1 depending on whether the difference is greater than or equal to the threshold value, but may take three or more values or a continuous value depending on the degree of the difference.

The center of gravity c obtained as described above can be regarded as the predicted central position of the tracked object in the current frame image based on the inter-frame difference image. Further, the center of gravity c may be referred to as the differential center of gravity below.

The weight determination unit 267 also obtains the sum sc of differences in the central region 1005 and the sum sb of differences in the peripheral region (S408, reference numeral 915 in FIG. 9). In FIG. 10C, the white pixels are the pixels with the difference in the central region, and the shaded pixels are the pixels with the difference in the peripheral region. Also here, the sums sc and sb of the differences may be obtained as weighted sums, and the setting of the weights is the same as above. Specifically, the sum of differences is obtained by the following formula (2).
Sum of differences=Σ _i (d(i)×w(i)) (2)
Here, the object of the summation (sigma) is the central region or the peripheral region.

Note that the sums sc and sb of the differences obtained by Equation (2) may be normalized by the areas of the central region and the peripheral region. In other words, sc and sb may be values obtained by dividing the values obtained by Equation (2) by the area of the central region or the peripheral region.

The weight determination unit 267 calculates the application rate (weight coefficient) e of the center of gravity with respect to the tentatively determined center position p based on sc and sb obtained in this way (S409). Here, the application rate e is calculated by the following formula.
e=f(sc+sb) _{0, α} ×f(sc/sb) _{β, 1} (3)
Here, as shown in FIG. 11, the function f(x) _{a, b} is 0 when x<a, 1 when x>b, and (x−a)/(b− a) is a function that takes

The calculation formula for e shown here is just an example, and other calculation formulas may be used if it is determined to take a value closer to 1 as the probability that the center of gravity c is the center position of the tracked object is higher. good too. In the above formula, α is a sufficiently small value (for example, 0.025), and the first term becomes 0 when sc+sb is less than or equal to α. Also, β is a relatively large value (for example, 0.5), and if sc/sb is less than or equal to β, the second term becomes 0. take a value. If the application rate e is determined in this way, the center of gravity of the difference c will be the center of the object to be tracked, as in the case where the background is moving and the difference is greater in the peripheral area than in the central area, or in the case where the obtained difference is small. When the probability of being at the position is low, the effect of correction by the differential center of gravity c can be set low, and inappropriate correction can be avoided.

The position correction unit 27 corrects the center position p by multiplying the position of the center of gravity c by the second sub-tracking unit 26 (center-of-gravity determination unit 266) by the application rate e with respect to the center position p by the first sub-tracking unit 25. (Correction) is performed (S410). The center position P (923) after correction is calculated by the following formula.
P=(1−e)×p+e×c (4)

Although the application rate e is determined according to the equation (3) based on the sums sc and sb of the differences in the adjusted difference image, the application rate e may be determined in consideration of other factors. .

For example, considering the reliability (probability) of the center position of the tracked object by the first sub-tracking unit 25, if the reliability is higher than the threshold, the application rate e is the value obtained by the above equation (3). It may be set to a value smaller than or set to zero. This threshold value is set to a value at which the position tracked by the first sub-tracking unit 25 can be regarded as sufficiently reliable. By doing so, it is possible to prevent the accuracy of the tracking position from deteriorating due to correction using the center of gravity based on the difference image between frames. For example, when the object to be tracked is stationary, the tracking position obtained by the first sub-tracking section 25 is sufficiently accurate. However, if there is a moving object in the vicinity, there is a possibility that the center of gravity based on the difference image between frames will be corrected inappropriately and the tracking accuracy will decrease. You can avoid this situation.

Further, if the difference between the center position p of the object tracked by the first sub-tracking unit 25 and the position of the differential center of gravity c is equal to or less than the threshold, the application rate e is set to a value smaller than the value obtained by the above equation (3). or zero. This threshold may be determined by experiments or the like, but it is assumed to be a sufficiently small value. Since the correlation filter is determined by accumulating past tracked images, even if the tracked position is slightly shifted from its actual position, the accumulation of inappropriate corrections will eventually lead to false positives. there is a possibility. Therefore, if the center position p obtained by the correlation filter and the center of gravity of difference c are sufficiently close, the result obtained by the correlation filter is emphasized and deterioration of the response of the correlation filter is avoided, thereby achieving accurate tracking.

As described above, the correction processing in step S305 is completed, and the center position P of the tracked object in the current frame image is determined.

Returning to the description of the flowchart in FIG. When the correction process of step S305 is completed as described above, the tracking unit 24 updates the center of the target area to the corrected center position P (S306). It also updates the size of the target region. Thus, after the tracking process is completed, the center of the target area is updated to the center position of the person to be tracked, and the size of the target area is also updated according to the tracking result. The update size of the target area may be estimated by a method using an image pyramid such as DSST (Discriminative Scale Space Tracking), or may be estimated based on the size of the target area in the previous frame, characteristics of lens distortion, camera viewpoint, and camera position. It may be determined based on the placement and/or the position of the target area in the image. The center of the target area after completion of the tracking process is the center position of the person to be tracked, and the foreground area in the target area is the existence area (bounding box) of the person to be tracked.

[Mask processing]
Next, the masking process (S406 in FIG. 7) of the other tracking target area during the correction process, which has been omitted from the description above, will be described. As described above, the masking process in step S406 is a process of setting a masking area in order to ignore (remove) differences caused by other tracked objects in the adjusted difference image 914 . FIG. 8 is a flowchart showing the details of mask processing S406. This process is repeated for each of the tracked objects other than the currently focused tracked object in the tracking process S106. In FIG. 12A, a person 1201 is the current tracked person, and a person 1202 is another tracked person. Therefore, the following steps S501 to S503 are repeated for each person 1202 .

Specifically, the other target region masking unit 264 acquires the target rectangle 1211 of the other tracked object 1202 in the previous frame image (S501), and changes the position and moving speed 1203 of the other tracked object 1202 up to the previous frame image. Based on this, the target rectangle 1211 of the other tracked object 1202 in the current frame image is predicted (S502). Then, the other target area masking unit 264 masks the area from the target area of the current frame image to either the rectangle 1210 or the rectangle 1211, and excludes it from being considered in the processing after step S407. The sum of the rectangle 1210 and the rectangle 1211 corresponds to the area in which an object other than the object of interest is expected to exist.

Under the situation where the persons to be tracked are dense or close to each other as shown in FIG. 12A, an adjusted difference image 914 as shown in FIG. 12C is obtained. By masking the region in which the other tracking target object 1202 is expected to exist and excluding it from the processing target, the masked adjusted difference image shown in FIG. used. By excluding the areas of other tracking objects in this way, it is possible to consider only the difference accompanying the movement of the tracking object of interest. The accuracy of the calculation of is improved.

<Advantageous effects of the present embodiment>
In this embodiment, in a human tracking device that uses a fisheye image without planar development, drift to the background can be suppressed and highly accurate human tracking can be realized. Drift is a tracking failure that occurs due to erroneous learning of features other than the tracked target during sequential learning. The object tracking algorithm by the first sub-tracking unit 25 learns local features including the neighboring area of the area where the object exists. Therefore, when the background changes intricately, a position different from the actual center of the object may be recognized as the center position. Such errors are accumulated by sequential learning, and the background may eventually be erroneously recognized as the tracking target. In this embodiment, the position of the object obtained by the first sub-tracking unit 25 is corrected by the position obtained based on the difference image between frames, so that the position of the object can be specified with higher accuracy. can be done. In particular, tracking of an object that changes drastically under a static and complex background, which is highly likely to be inappropriately tracked by the correlation filter without impairing the tracking performance of the correlation filter of the first sub-tracking unit 25 as much as possible. can be performed with high accuracy.

Further, in the conventional background subtraction method, since the current frame image and the frame images before and after it are used for processing, the tracking result for the current frame image is obtained after the image one frame after is input. However, in this embodiment, since only the current frame image and the previous frame image are used, the effect can be obtained immediately from the start of tracking. In addition, the conventional background subtraction method requires a large area to be processed, which results in a high computational cost and is not suitable for embedded devices with few computational resources. Since the target is an image and the calculation cost is low, it can be suitably applied to an embedded device.

In addition, by masking areas in which other tracked persons are expected to exist and processing them, when the tracked persons are close or close to each other, the difference center of gravity is affected by the other tracked persons. You can avoid situations where you don't want it right. Even if all the difference information is excluded by the masking process, correction based on the difference centroid is simply not performed, and the tracking position using the correlation filter can be used. That is, there is an advantage that the mask processing does not adversely affect the tracking result.

In addition, since the processing load required to obtain the difference image between frames and the information necessary for correcting the center position is small, the method can be suitably applied to embedded devices with few computational resources.

<Others>
The above-described embodiment is merely an example of the configuration of the present invention. The present invention is not limited to the specific forms described above, and various modifications are possible within the technical scope of the present invention.

Also, in the above embodiment, the first sub-tracking unit 25 performs tracking processing using a correlation filter, but tracking may be performed using other algorithms. For example, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long
Tracking may be performed using a deep learning model such as Short-Term Memory) or a pattern recognition model such as SVM (Support Vector Machine). Also, the second sub-tracking unit 26 may adopt a technique other than the above as long as it is a moving object detection technique based on the difference.

Further, in the above embodiment, the fisheye image is processed without planar development, but the image processed by planarizing the fisheye image may be processed, or the image captured by a normal camera may be processed. good too.

<Appendix>
(1) acquisition means (21) for acquiring the position of the object in the first frame image;
tracking means (24) for obtaining the position of the object from a second frame image, which is a frame image after the first frame image;
An object tracking device (1) comprising
Said tracking means (24) are:
a first sub-tracking means (25) for obtaining a first coordinate (p) of the object in the second frame image by a first tracking algorithm based on the feature amount extracted from the second frame image;
a second sub-tracking means (26) for obtaining a second coordinate (c), which is the center of gravity of a moving area, based on an inter-frame difference image (913) between said first frame image and said second frame image; ,
Position specifying means (27) for obtaining a position (P) of the object in the second frame image based on the first coordinate (p) and the second coordinate (c);
comprising
An object tracking device (1) characterized by:

(2) an acquisition step (S102) of acquiring the position of the object in the first frame image;
a tracking step (S106) for obtaining the position of the object from a second frame image, which is a frame image after the first frame image;
An object tracking method comprising:
The tracking step includes:
obtaining a first coordinate (p) of the object in the second frame image by a first tracking algorithm based on the feature amount extracted from the second frame image (S304);
a step (S408) of obtaining a second coordinate (c), which is the center of gravity of an area with motion, based on an inter-frame difference image (913) between the first frame image and the second frame image;
obtaining a position (P) of the object in the second frame image based on the first coordinates and the second coordinates (S410);
An object tracking method comprising:

1: Person tracking device 2: Monitoring system 10: Fisheye camera 11: Tracking target area 12: Ceiling 13: Person

Claims (12)

acquisition means for acquiring the position of the object in the first frame image;
tracking means for obtaining the position of the object from a second frame image, which is a frame image after the first frame image;
An object tracking device comprising:
The tracking means are
a first sub-tracking means for obtaining a first coordinate of the object in the second frame image by a first tracking algorithm based on the feature quantity extracted from the second frame image;
a second sub-tracking means for obtaining a second coordinate, which is the center of gravity of a moving area, based on an inter-frame difference image between the first frame image and the second frame image;
position specifying means for determining the position of the object in the second frame image based on the first coordinates and the second coordinates;
with
The second sub-tracking means,
generating a likelihood map representing the probability that the object exists in the second frame image using a color histogram of the area containing the object generated using at least the first frame image;
generating an adjusted difference image by multiplying the difference image between the frames of the first frame image and the second frame image by the likelihood map;
Obtaining the center of gravity of the pixel region based on the degree of difference as the second coordinate in the adjusted difference image;
An object tracking device characterized by: The second sub-tracking means,
setting a central region centered on a position where the object is assumed to exist and a peripheral region around the central region for the adjusted difference image;
Giving different weights to pixels in the central region and pixels in the peripheral region to obtain the center of gravity;
The object tracking device according to claim 1 . The second sub-tracking means,
setting a central region centered on a position where the object is assumed to exist and a peripheral region around the central region for the adjusted difference image;
determining a weighting factor based on the sum of differences in the central region and the sum of differences in the peripheral region;
The position specifying means obtains coordinates determined as a weighted average of the first coordinates and the second coordinates using the weighting factor as the position of the object.
3. An object tracking device according to claim 1 or 2 . The second sub-tracking means obtains the sum of the center of gravity and the difference by excluding an area in the adjusted difference image in which an object other than the object of interest is expected to exist.
4. An object tracking device according to claim 3 . The position where the object is presumed to exist is determined based on the position of the object in the first frame image and the moving speed of the object in the first frame image,
An object tracking device according to any one of claims 2 to 4 . When the likelihood of the first coordinate obtained by the first sub-tracking means is equal to or greater than a second threshold, the position specifying means weights the second coordinate less than otherwise, determining a coordinate determined as a weighted average of the first coordinate and the second coordinate as the position of the object;
An object tracking device according to any one of claims 1 to 5 . When the difference between the first coordinate and the second coordinate is less than a third threshold, the position identifying means weights the second coordinate less than otherwise, and determines the position of the first coordinate. determining a coordinate determined as a weighted average of the coordinate and the second coordinate as the position of the object;
An object tracking device according to any one of claims 1 to 6 . The first tracking algorithm is an algorithm for determining the position of the object in the second frame image by a correlation filter based on the feature amount obtained from the vicinity of the object in the first frame image.
An object tracking device according to any one of claims 1 to 7 . The first frame image and the second frame image are fisheye images obtained by a fisheye camera,
The object tracking device according to any one of claims 1 to 8 , characterized in that: an object tracking device according to any one of claims 1 to 9;
a fisheye camera,
surveillance system. an acquisition step of acquiring the position of the object in the first frame image;
a tracking step of obtaining the position of the object from a second frame image, which is a frame image after the first frame image;
An object tracking method comprising:
The tracking step includes:
determining first coordinates of the object in the second frame image by a first tracking algorithm based on the feature quantity extracted from the second frame image;
obtaining second coordinates, which are the center of gravity of a moving region, based on an inter-frame difference image between the first frame image and the second frame image;
determining the position of the object in the second frame image based on the first coordinates and the second coordinates;
including
In the step of determining the second coordinates,
generating a likelihood map representing the probability that the object exists in the second frame image, using the color histogram of the area containing the object generated using at least the first frame image;
generating an adjusted difference image by multiplying the difference image between the frames of the first frame image and the second frame image by the likelihood map;
Obtaining the center of gravity of the pixel region based on the degree of difference as the second coordinate in the adjusted difference image;
An object tracking method characterized by: A program for causing a computer to perform the steps of the method according to claim 11.

Images