JP2020149641A - Object tracking device and object tracking method - Google Patents

Abstract

To provide an object tracking device that reduces drift to a background to perform accurate object tracking.SOLUTION: In a monitoring system 2, a human tracking device 1 has an image input unit that acquires the position of an object in a first frame image, and a tracking unit that determines the position of the object from a second frame image. The tracking means includes: a feature quantity extraction unit that extracts feature quantities from a target area of the second frame image; a likelihood map creation unit that determines, based on the feature quantities, a likelihood map representing the likelihood of the presence of the object for the target area of the second frame image; and a position specification unit that, when the likelihood map has one peak, specifies the position of the peak as the position of the object, and when the likelihood map has a plurality of peaks, specifies, as the position of the object, the position of a peak selected in consideration of image similarity representing the similarity between an image area in the vicinity of the position of the object in the first frame image and image areas in the vicinity of the peaks in the second frame image.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for tracking an object in a moving image.

Object tracking, which tracks an object detected in a frame with a moving image (time series image), is an important technique in the field of computer vision.

The background subtraction method, which is a general tracking method, is lost when the movement of the tracking target stops. For example, if the tracking target is a person, it will be lost if this person sits in a chair, so it is not suitable for monitoring in the office. Further, in template matching, when an object is deformed and the difference from the template exceeds a predetermined threshold value, the object is lost. In the case of a person, tracking fails because the movement of the person causes a large deformation compared to the template.

On the other hand, in Non-Patent Document 1, the position of the tracking target is determined based on the composite likelihood obtained by combining the likelihood based on the luminance gradient (HOG feature amount) and the likelihood based on the color feature (color histogram). .. It has been reported that robust tracking is possible by performing tracking using features related to shape and color in a complementary manner.

Further, Patent Document 1 discloses that a scene change is detected, and a feature amount having an optimum tracking performance for the changed scene is selected and tracked.

By the way, in the fields of building automation (BA) and factory automation (FA), image sensors automatically measure the "number", "position", "flow line", etc. of people, and optimally control equipment such as lighting and air conditioning. Application is needed. In such applications, in order to acquire image information in the widest possible range, it is called an ultra-wide-angle camera (fisheye camera, omnidirectional camera, omnidirectional camera, etc.) equipped with a fisheye lens (fisheye lens). The meaning is the same. In this specification, the term "fisheye camera" is used). Further, in the above application, in order to acquire image information in as wide a range as possible, a camera mounted on a high place such as a ceiling is arranged so that the viewpoint of the camera is the top view. In the camera of this arrangement, the viewpoint for photographing the person is the front view when the person is around the image and the top view when the person is in the center of the image.

The image taken by the fisheye camera is deformed due to the distortion of the appearance of the object to be photographed depending on the position in the photographing surface. Furthermore, when the viewpoint of the camera is set to the top view, the appearance changes depending on the position of the tracking target. Further, in an environment with limited processing capacity such as an embedded device, the frame rate is considered to be low, and there is a peculiarity that the amount of movement of an object and the amount of features change greatly between frames. Therefore, the conventional tracking method may not be able to track accurately.

JP-A-2015-79502

Bertinetto, Luca, et al. "Staple: Complementary learners for real-time tracking." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an object tracking technique with higher accuracy than before.

In order to achieve the above object, the present invention adopts the following configuration.

The first aspect of the present invention is a tracking for obtaining the position of an object from an acquisition means for acquiring the position of the object in the first frame image and a second frame image which is a frame image after the first frame image. An object tracking device including means, wherein the tracking means relates to a feature amount extracting means for extracting a feature amount from a target area of the second frame image and the target area of the second frame image. A likelihood calculation means for obtaining a likelihood map representing the certainty that an object exists based on the feature amount, and when there is one peak in the likelihood map, the position of the peak is determined by the object. When it is specified as a position and there are a plurality of peaks in the likelihood map, an image area near the position of the object in the first frame image and an image area near each peak in the second frame image. Provided is an object tracking device including a position determining means for specifying the position of a peak selected in consideration of the image similarity representing the similarity as the position of the object.

The object to be tracked, that is, the "object" may be any object, and examples thereof include a human body, a face, an animal, and a vehicle. The "target area" is an area for searching for an object in the second frame image, and is typically a partial area determined based on the position of the object in the first frame image. The "image similarity" is an index showing the similarity between images, and is evaluated by, for example, the difference in average color and average brightness in a region. It is preferable that the image area near the position of the object in the first frame image and the image area near the position of the peak in the second frame image have the same feature amount, and the object (foreground) It is preferably a part of the area, and particularly preferably a part of the center of the object (foreground).

The likelihood map obtained by the likelihood calculation unit is expected to take the maximum value at the position where the object exists, but may take the maximum value at the position of the object different from the object. Therefore, simply determining the position of the maximum value on the likelihood map as the position of the tracking object causes a tracking error called transfer (drift). Therefore, in the present invention, when a plurality of peaks (local peaks) exist in the likelihood map, the peaks are selected in consideration of the image similarity near the position of the object, and the selected peak positions are targeted. Identify as the location of an object. By selecting the peak in consideration of the image similarity in this way, the tracking accuracy is improved.

For example, when there are a plurality of peaks in the likelihood map, the positioning means of the present invention targets the position of the peak having the maximum image similarity among the peaks having the likelihood value equal to or higher than the threshold value. It may be specified as the position of an object. At this time, the above threshold value may be determined for each peak according to the image similarity.

The method of obtaining the likelihood map by the likelihood calculation means of the present invention is not particularly limited, but for example, pay attention to the first feature amount which is a feature amount related to shape and the second feature amount which is a feature amount related to color or brightness. You may find a map of likelihood. Examples of the feature amount related to the shape include at least one of the HOG feature amount, the LBP feature amount, the SHIFT feature amount, and the SURF feature amount. Color features include at least one of color histograms, luminance histograms, and Color Names features. The likelihood calculation means of the present invention may obtain a first likelihood based on the first feature amount and a second likelihood based on the second feature amount, and generate a composite likelihood map by synthesizing these.

In the present invention, the image similarity considered at the time of peak selection is, for example, at least one of the difference in image information including at least one of the average color, the average brightness, and the representative color, the absolute value of the difference, and the square of the difference. It can be decided based on. Further, the image similarity is determined based on at least one of the feature amount related to the shape such as HOG or the feature amount related to the color such as the color histogram, at least one of the histogram intersection, the butterfly coefficient, and the Earth Mover's Distance. be able to. In addition, image similarity can be determined by template matching. Further, a method of measuring the degree of difference based on at least one of the sum of squares of the differences and the sum of the absolute values of the differences can be adopted instead of the degree of similarity. Image similarity is a measure that allows you to grasp how similar two images are, and may be an index with a larger value as they are similar, such as a histogram intersection, or an absolute value of difference. The index may have a smaller value as it is similar to.

Further, the image to be processed in the present invention may be a fisheye image obtained by a fisheye camera. A "fisheye camera" is a camera equipped with a fisheye lens, which is capable of shooting at an ultra-wide angle compared to a normal camera. Omnidirectional cameras, spherical cameras, and fisheye cameras are all types of ultra-wide-angle cameras, and they all have the same meaning. The fisheye camera may be installed so as to look down on the detection target area from above the detection target area. Typically, the optical axis of the fisheye camera is installed so as to face vertically downward, but the optical axis of the fisheye camera may be tilted with respect to the vertical direction. Since the fisheye image has a large distortion, the characteristic change of the object between frames is large especially in the image with a low frame rate, and the drift to the background occurs frequently. Furthermore, if the optical axis of the camera is installed so that it faces vertically downward, the viewpoint at which the object is photographed changes depending on the position of the object in the image. Therefore, especially in a low frame rate image, the object is greatly deformed and tracking fails. Occurs frequently. However, according to the present invention, even in such a fisheye image, accurate tracking is possible even if the optical axis of the camera is installed so as to face vertically downward. However, the image to be processed by the present invention is not limited to the fisheye image, and may be a normal image (an image with less distortion or an image with a high frame rate).

The second aspect of the present invention is a tracking for obtaining the position of the object from the acquisition step of acquiring the position of the object in the first frame image and the second frame image which is a frame image after the first frame image. An object tracking method including a step, wherein the tracking step relates to a feature amount extraction step for extracting a feature amount from a target area of the second frame image and the target area of the second frame image. A likelihood calculation step for obtaining a likelihood map representing the certainty that an object exists based on the feature amount, and when there is one peak in the likelihood map, the position of the peak is determined by the object. When it is specified as a position and there are a plurality of peaks in the likelihood map, an image area near the position of the object in the first frame image and an image area near each peak in the second frame image. Provided is an object tracking method comprising a positioning step of identifying the position of a peak selected in consideration of the image similarity representing the similarity of the object as the position of the object.

The present invention may be regarded as an object tracking device having at least a part of the above means, or may be regarded as an image processing device or a monitoring system. Further, the present invention may be regarded as an object tracking method, an image processing method, and a monitoring method including at least a part of the above processing. Further, the present invention can also be regarded as a program for realizing such a method and a recording medium in which the program is recorded non-temporarily. The present invention can be constructed by combining each of the above means and treatments with each other as much as possible.

According to the present invention, object tracking can be performed with higher accuracy than before.

FIG. 1 is a diagram showing an application example of the person tracking device according to the present invention. FIG. 2 is a diagram showing a configuration of a monitoring system including a person tracking device. FIG. 3 is a flowchart of the entire process performed by the person tracking device. FIG. 4 is a flowchart of the learning process. FIG. 5 is a flowchart of the tracking process. FIG. 6 is a flowchart of the peak selection process in the tracking process. FIG. 7 is a diagram illustrating map generation of synthetic likelihood in the tracking process. FIG. 8 is a diagram showing an example of a map of composite likelihood. FIG. 9 is a diagram illustrating a peak selection process when there are a plurality of peaks in the composite likelihood map.

<Application example>
An application example of the object tracking device according to the present invention will be described with reference to FIG. The person tracking device 1 analyzes the fisheye image obtained by the fisheye camera 10 installed above the tracking target area 11 (for example, the ceiling 12), and detects the person 13 existing in the tracking target area 11.・ It is a tracking device. The person tracking device 1 detects, recognizes, and tracks a person 13 passing through the tracking target area 11, for example, in an office or a factory. In the example of FIG. 1, the region of each of the four human bodies detected from the fisheye image is shown by a bounding box. The detection result of the person tracking device 1 is output to an external device, and is used for, for example, counting the number of people, controlling various devices such as lighting and air conditioning, monitoring a suspicious person, and analyzing a flow line.

Object tracking is performed by specifying the position of an area having the same characteristics as the object, targeting the target area (target area) of the current frame near the position of the object specified in the previous frame image. Here, a plurality of likelihood peaks representing the object-likeness may appear in the target region. In such a case, the human tracking device 1 does not simply specify the peak with the highest likelihood as the position of the object, but the average color near the center position of the object in the previous frame image and the current frame. The peak that minimizes the difference from the average color in the vicinity of the peak position is determined as the position of the object. By specifying the peak, that is, the position of the object in consideration of the average color in this way, drift to the background can be suppressed, and accurate tracking becomes possible. Further, since the calculation of the average color is a process with a relatively light calculation load, high-speed tracking can be realized.

<Monitoring system>
An embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration of a monitoring system to which the person tracking device according to the embodiment of the present invention is applied. The surveillance system 2 includes a fisheye camera 10 and a person tracking device 1.

The fisheye camera 10 is an image pickup device having an optical system including a fisheye lens and an image sensor (an image sensor such as a CCD or CMOS). As shown in FIG. 1, the fisheye camera 10 is installed on the ceiling 12 of the tracking target area 11 with the optical axis facing vertically downward, and images of the tracking target area 11 in all directions (360 degrees) are captured. You should take a picture. The fish-eye camera 10 is connected to the person tracking device 1 by wire (USB cable, LAN cable, etc.) or wirelessly (WiFi, etc.), and the image data captured by the fish-eye camera 10 is taken into the person tracking device 1. The image data may be either a monochrome image or a color image, and the resolution, frame rate, and format of the image data are arbitrary. In this embodiment, it is assumed that a color (RGB) image captured at 10 fps (10 images per second) is used.

The person tracking device 1 of the present embodiment includes an image input unit 20, a human body detection unit 21, a learning unit 22, a storage unit 23, a tracking unit 24, and an output unit 28.

The image input unit 20 has a function of capturing image data from the fisheye camera 10. The captured image data is delivered to the human body detection unit 21 and the tracking unit 24. This image data may be stored in the storage unit 23.

The human body detection unit 21 has a function of detecting a human body from a fisheye image by using an algorithm for detecting the human body. The human body detected by the human body detection unit 21 is the target of the tracking process by the tracking unit 24. The human body detection unit 21 may detect only a person newly appearing in the image, or may exclude the vicinity of the position where the person to be tracked exists from the target of the detection process. Further, a Tracking-by-detection method may be adopted in which the human body detection unit 21 detects a person in the entire image at a fixed time interval or frame interval, and then the tracking unit 24 performs tracking processing.

The learning unit 22 learns the characteristics of the human body to be tracked from the image of the human body detected by the human body detecting unit 21 or specified by the tracking unit 24, and stores the learning result in the storage unit 23. Here, the learning unit 22 obtains a correlation filter for performing evaluation based on shape features, a color histogram for performing evaluation based on color features, and an average color at the center position. The learning unit 22 performs learning every frame, and updates the learning result obtained from the current frame by reflecting it in the past learning result with a predetermined coefficient.

The storage unit 23 stores the learning result learned by the learning unit 22. The storage unit 23 also stores hyperparameters of learning processing and tracking processing, such as feature amounts to be used, parameters of each feature amount, learning coefficients, weighting coefficients at the time of synthesis, and initial values of threshold values in peak selection.

The tracking unit 24 identifies the position of the person to be tracked in the current frame image. Initially, the tracking unit 24 uses a region including a detection position by the human body detection unit 21 as a target region, and identifies an object position having the same characteristics as a person detected in the target region. After that, the position of the person to be tracked is specified from the current frame image with the vicinity of the position specified by the tracking unit 24 for the previous frame image as the target area.

The feature amount extraction unit 25 extracts the feature amount related to the shape of the object and the feature amount related to the color from the target region. The feature amount extraction unit 25 extracts the HOG feature amount as a feature related to the shape, and extracts the color histogram as the feature amount related to the color.

The likelihood map generator 26 uses the extracted features and the correlation filter and color histogram stored in the storage unit 23 to indicate the likelihood that the tracked object exists at each position in the target area. Generate a map of degrees. The likelihood map generation unit 26 generates a composite likelihood map in which the likelihood based on the shape feature and the correlation filter and the likelihood based on the color feature and the color histogram are combined. The likelihood map is also called a response map.

The position specifying unit 27 identifies the position of the tracking object in the current frame image based on the map of the composite likelihood. Specifically, the position specifying unit 27 specifies the position as the position of the tracking object when there is one peak in the map of the composite likelihood. On the other hand, in the position specifying unit 27, when there are a plurality of peaks, the difference between the average color near the center position of the object in the previous frame image and the average color near the center of the peak position in the current frame image is the smallest peak. The position is specified as the position of the tracked object. The minimum difference in average color means, in other words, the maximum image similarity based on the average color.

The output unit 28 has a function of outputting information such as a fisheye image, a detection result, and a tracking result to an external device. For example, the output unit 28 may display information on a display as an external device, transfer information to a computer as an external device, or for a lighting device, an air conditioner, or an FA device as an external device. Information and control signals may be transmitted.

The person tracking device 1 can be configured by, for example, a computer including a CPU (processor), a memory, a storage, and the like. In that case, the configuration shown in FIG. 2 is realized by loading the program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or an embedded computer such as an onboard computer. Alternatively, all or part of the configuration shown in FIG. 2 may be configured by ASIC, FPGA, or the like. Alternatively, all or part of the configuration shown in FIG. 2 may be realized by cloud computing or distributed computing.

<Overall processing>
FIG. 3 is an overall flowchart of the person tracking process by the monitoring system 2. The overall flow of the person tracking process will be described with reference to FIG.

First, in step S101, the user sets the learning and tracking hyperparameters for the person tracking device 1. Examples of hyperparameters include features to be used, parameters of each feature, learning coefficient, weighting coefficient at the time of synthesis, initial value of threshold value in peak selection, and the like. The input hyperparameters are stored in the storage unit 23.

Next, in step S102, the person tracking device 1 acquires the target area. The target area is an area that includes the area in which the person to be tracked exists and its surroundings, and is an area in which the person to be tracked is likely to exist. It can be said that the target area is an area to be processed by the tracking unit 24. In the present embodiment, the initial position of the person to be tracked is detected by the human body detection unit 21. However, the initial position of the tracked person may be input by the user, for example.

Hereinafter, the processes of steps S104 to S107 are repeatedly performed. When the end condition is satisfied in the end determination in step S103, the process ends. The end condition can be, for example, the loss of the tracked person (frame out) or the end of the moving image.

In step S104, the image input unit 20 inputs one frame of fisheye image from the fisheye camera 10. At this time, a plane-developed image in which the distortion of the fisheye image is corrected may be created and the subsequent processing may be performed, but in the monitoring system 2 of the present embodiment, the fisheye image is detected and tracked as it is (while being distorted). Used for processing.

In step S105, it is determined whether or not the current frame is the first image. Here, the first image is a frame image in which the initial position of the tracking target person is given, and is typically a frame image in which the tracking target person is detected by the human body detection unit 21.

If the current frame is an image of a frame after the first image, the process proceeds to step S106, and the tracking unit 24 executes the tracking process. The details of the tracking process will be described later.

In step S107, the learning unit 22 executes the learning process based on the area where the target person exists in the current frame image. The details of the learning process will be described later.

In this way, the position of the person to be tracked by the tracking process S106 is specified every frame, and the tracking is realized. Further, the tracking method of the present embodiment employs a sequential learning type tracking algorithm that learns the characteristics of the person to be tracked every frame.

<Learning process>
FIG. 4 is a flowchart showing the details of the learning process in step S107. Further, FIG. 7 is a diagram illustrating a learning process and a tracking process using the learning result. Hereinafter, the learning process will be described with reference to FIGS. 4 and 7.

First, the learning unit 22 cuts out the target area 74 from the current frame image (S201). As shown in FIG. 7, the target area 74 is an area including a foreground area 72 and a background area 73 of a person. The foreground area 72 is an area where the tracked person exists, and the background area is an area where the tracked person does not exist. The size of the background area 73 is determined according to the size of the foreground area 72. For example, the size of the background region 73 is determined so that the size of the foreground region 72 is a predetermined ratio (for example, 1/3) of the total size of the target region 74. Since the target area is updated at the end of the tracking process so that the center is the position of the tracking target person (step S308 in FIG. 5), the center of the target area 74 is equal to the center position of the tracking target person.

The learning unit 22 extracts the average color near the center position 71 of the target area 74 and stores it in the storage unit 23 (S202). Here, the vicinity of the center position 71 is a region smaller than the foreground region 72 including the center position 71, and is typically a rectangular region centered on the center position 71. The size of this neighborhood region may be a fixed size (for example, 3 × 3) or a size corresponding to the size of the foreground region 72 (for example, half the size).

The learning unit 22 also acquires the HOG feature amount in the target region 74 (S203). The HOG feature amount is a feature amount obtained by histogramizing the brightness gradient direction of the local region, and can be regarded as a feature amount representing the shape and contour of the object. Here, the HOG feature amount is adopted, but other feature amounts representing the shape / contour of the object, for example, the LBP feature amount, the SHIFT feature amount, and the SURF feature amount may be adopted.

The learning unit 22 obtains a correlation filter 76 such that the response has a peak at the center of the target (S204). Specifically, after extracting the HOG feature amount, the correlation filter 76 can be obtained by finding a filter that is closest to the ideal response having a peak only in the center with respect to the correlation of the feature amount itself. When the correlation filter is calculated in Fourier space, the feature quantity may be multiplied by the window function. The HOG feature amount is stored in the storage unit 23 for use when applying a correlation filter in the tracking process of the next frame.

The learning unit 22 also acquires the color histogram 77 in the target region 74 (S205). Specifically, the color histograms of the foreground region 72 and the background region 73 are acquired. The color histogram is a feature amount representing a color, and the Color Names (CN) feature amount can be adopted as another feature amount representing a color. Further, the luminance histogram may be adopted as the feature quantity representing the luminance feature instead of the color feature quantity.

If this learning is the first learning (S206-YES), the correlation filter and the color histogram generated in steps S203 and S205 are stored in the storage unit 23 as they are. On the other hand, if the current learning is the second and subsequent learning (S206-NO), the process proceeds to step S207.

In step S207, the learning unit 22 obtains a new correlation filter by synthesizing the correlation filter obtained last time (correlation filter stored in the storage unit 23) and the correlation filter obtained in step S204 this time, and the storage unit 23 obtains a new correlation filter. Remember in. Further, in step S208, the learning unit 22 obtains a new color histogram by synthesizing the color histogram obtained last time (the color histogram stored in the storage unit 23) and the color histogram obtained in step S205 this time. It is stored in the storage unit 23. The weight (learning coefficient) at the time of synthesis may be appropriately determined.

<Tracking process>
FIG. 5 is a flowchart showing the details of the tracking process in step S106. Further, FIG. 7 is a diagram illustrating a learning process and a tracking process using the learning result. Hereinafter, the tracking process will be described with reference to FIGS. 5 and 7.

The tracking unit 24 cuts out the target area 75 from the current frame image (S301). Since the target area is updated at the end of the previous tracking process so that the center is the position of the tracking target person (step S308 in FIG. 5), the center of the target area 74 is equal to the center position of the tracking target person. .. In FIG. 7, when the processing target of the tracking unit 24 is the image in the T + 1th frame, the target area 75 corresponding to the target area 74 centered on the position of the tracking target person specified in the T frame is cut out.

The feature amount extraction unit 25 extracts the HOG feature amount from each cell in the target area 75 (S302). The likelihood map generation unit 26 applies a correlation filter 76 to the correlation between the HOG feature amount in the target area 75 and the HOG feature amount stored in the storage unit 23 to obtain a likelihood map 78 (response map). Find (S303). Graph 81 of FIG. 8A and graph 84 of FIG. 8B are examples of a map 78 of likelihood based on HOG features. The likelihood map 81 is a map showing the certainty (likelihood) of the person to be tracked for each position in the target area 75.

From the color of each pixel in the target area 75 and the color histogram 77 stored in the storage unit 23, the likelihood map generation unit 26 ensures that each cell in the target area 75 is a tracking target person (foreground). A likelihood map 79 (response map) representing the likelihood (likelihood) is generated. More specifically, the likelihood map generation unit 26 obtains the foreground likelihood of the pixel of interest based on the color histogram 77 stored in the storage unit 23 and the color of the pixel of interest. Then, by taking the average of the foreground likelihoods of the pixels included in each cell, the likelihood that the cell is the person to be tracked can be obtained. Graph 82 of FIG. 8A and graph 85 of FIG. 8B are examples of the likelihood map 78 based on the color histogram.

The likelihood map generation unit 26 synthesizes the likelihood map 78 based on the correlation filter 76 obtained as described above and the likelihood map 79 based on the color histogram 77 to generate the composite likelihood map 80. (The method of synthesis is not particularly limited, and the two likelihoods may be simply averaged or weighted and averaged. Graph 83 in FIG. 8A and graph 86 in FIG. 8B show the composite likelihood. This is an example of map 80 (composite response map).

The position specifying unit 27 selects one peak from the map of the composite likelihood and determines that the peak position is the center position of the tracked person in the current frame image (S307). Here, as shown in FIG. 8A, when the map of the composite likelihood has only one peak, it can be said that the position of the peak is the position of the person to be tracked. However, as shown in FIG. 8B, when the composite response map has a plurality of peaks, simply selecting the peak with the highest value (composite likelihood) may cause drift and mistracking. Therefore, the position specifying unit 27 realizes highly accurate tracking by selecting the peak by the process shown in the flowchart of FIG.

An outline of the processing performed by the flowchart of FIG. 6 will be briefly described with reference to FIG. In FIG. 9, the T frame is the front frame, and the T + 1 frame is the current frame. The image 91 represents the target area in the previous frame image, and the center 92 thereof is the center of the position where the target person exists. The image 94 represents a target region in the current frame image, in which a plurality of peaks 95 are extracted.

Of the plurality of peaks 95 extracted in the current frame image, the position specifying unit 27 has the average color in the region 93 near the center position 92 of the tracked person in the previous frame image and the region near the peak 95 in the current frame image. Select the peak with the smallest difference from the average color at 96.

In the following, for the sake of simplification of the explanation, the "average color in the area near the center position of the person to be tracked in the previous frame image" is referred to as the "center average color of the previous frame image", and the "current frame image". The "average color in the region near the peak position of the current frame image" is referred to as the "peak position average color of the current frame image".

Hereinafter, a more detailed description will be given with reference to FIG. The positioning unit 27 extracts a local peak from the composite likelihood map (S401). The local peak can be said to be the position where the maximum value is taken in the map of the composite likelihood. The local peak may be extracted, for example, by determining whether or not the value of the target pixel is equal to or greater than the value of the neighboring pixel. The processing after step S402 is performed for each of the local peaks detected here.

The position specifying unit 27 extracts the average color in the region 96 near the peak 95 of the current frame image 94 and temporarily stores it (S402). The method of obtaining the average color is not particularly limited.

The position specifying unit 27 determines whether or not the peak currently being processed is the first peak (S403), and if it is the first peak, proceeds to step S408 and selects this peak. The selection here is a tentative selection, and the peak selected after exiting the loop processing is the final selection result. The position specifying unit 27 stores the position of the selected peak in the storage unit 23. Further, the position specifying unit 27 stores the difference between the peak position average color of the current frame image and the center average color of the previous frame image in the storage unit 23.

If the peak currently being processed is not the first peak, processing proceeds to step S404. The position specifying unit 27 determines whether or not the peak value is equal to or higher than the threshold value A (S404). This threshold value A may be a fixed value given by presetting, or may be a value updated every time the tracking process of each frame is performed.

If the peak value is equal to or higher than the threshold value A (S404-YES), the position specifying unit 27 corrects the threshold value according to the difference between the center average color of the previous frame image and the peak position average color of the current frame image (step). S405). The corrected threshold value is referred to as threshold value B. Specifically, the larger the difference in average color, the smaller the threshold value, and the smaller the difference in average color, the larger the threshold value. The threshold correction is performed in order to improve the robustness to lighting changes between frames.

The position specifying unit 27 determines whether or not the value at the peak to be processed is equal to or higher than the correction threshold value B (S406). If the peak value is equal to or higher than the correction threshold value B, the position specifying unit 27 further determines that the difference between the center average color of the previous frame image and the peak position average color of the current frame image is the peak value of the correction threshold value B or higher. It is determined whether it is the smallest among them (S409). This determination may be replaced by the determination that the difference in average color at the current peak is less than the difference in average color at the tentatively selected peak. If the difference in average color is the smallest (S409-YES)
), The position specifying unit 27 selects the peak to be processed, and stores the peak position and the average color difference in the storage unit 23.

On the other hand, if the value at the peak to be processed is less than the threshold value A in the determination in step S404, or if the value at the peak to be processed is less than the correction threshold B in the determination in step S406, the process proceeds to step S407. .. The position specifying unit 27 determines in step S407 whether the value of the peak is the maximum so far, and if it is the maximum, selects this peak in step S408.

By performing the above processes of steps S402 to S410 for all the peaks extracted in step S401, the peak value is equal to or more than the threshold value, and the center average color of the previous frame image and the current frame image Peak position The peak with the smallest difference from the average color is selected.

In the above flowchart, whether or not the peak value is equal to or higher than the threshold value A is determined in step S404, but this process is omitted and only the determination (S406) based on the correction threshold value B is performed. You may. Further, when the peak value is below the threshold value (S404-NO, S406-NO), if the peak value is the maximum so far, it is selected (S407-S408), but it is not selected regardless of the peak value. You may do so. This is because it is assumed that the average color difference is large for the peak whose peak value is less than the threshold value, and then another peak is selected (S410). Further, although the robustness against lighting change is improved by the threshold correction process (S404) based on the average color difference, this process may be omitted and a fixed threshold value may be used.

Returning to the description of the flowchart of FIG. When the peak selection process in step S307 is completed as described above, the position specifying unit 27 updates the center of the target region to the position of the selected peak (S308), and updates the size of the target region (S309). In this way, after the tracking process is completed, the center of the target area is updated to the center position of the tracked person, and the size of the target area is also updated according to the tracking result. The update size of the target area may be estimated by a method using an image pyramid such as DSST (Discriminative Scale Space Tracking), or the size of the target area in the previous frame, the characteristics of lens distortion, the viewpoint of the camera, and the camera. It may be determined based on at least one of the placement and the position of the target area in the image. The center of the target area after the completion of the tracking process is the center position of the tracking target person, and the foreground area in the target area is the existing area (bounding box) of the tracking target person.

<Advantageous effect of this embodiment>
In the present embodiment, in a person tracking device that uses a fisheye image without developing a plane, it is possible to suppress drift to the background and realize highly accurate person tracking. Drift is a tracking failure that occurs due to erroneous learning of features other than the tracked object during sequential learning. It occurs when there is an object (background) similar to the person to be tracked in the image, under a complicated background, or when there is an occlusion. In general, when there is an object (background) similar to the tracked person, multiple peaks appear in the likelihood map in the presence of complex backgrounds and occlusions. Then, in such a case, if a peak corresponding to an object other than the tracking target is erroneously selected, drift occurs. In the present embodiment, when a plurality of peaks appear in the composite likelihood map, the peaks are selected in consideration of the average color at the center position, instead of simply selecting the peak with the maximum likelihood. This makes it possible to erroneously select a peak corresponding to an object other than the tracked object, that is, to reduce the occurrence of drift. If the occurrence of drift can be reduced, errors in the tracking result are reduced, and highly accurate tracking can be realized.

Further, since the calculation of the average color is a process in which the calculation load and the amount of memory used are relatively small, the method of the present embodiment can be realized even with an embedded device having few computational resources.

<Others>
The above-described embodiment is merely an example of a configuration example of the present invention. The present invention is not limited to the above-mentioned specific form, and various modifications can be made within the scope of its technical idea.

For example, in the above embodiment, when a plurality of peaks appear in the composite likelihood map, the peak to be selected is determined in consideration of the average color. However, the peak having the highest image similarity between the region near the center position of the tracking object of the previous frame image and the region near the peak position of the current frame image may be selected. As a method for evaluating image similarity other than the average color, for example, at least one of the difference, the absolute value of the difference, and the square of the difference, using at least one of the image information represented by the scalar such as the average brightness and the representative color as a feature amount. A method can be adopted in which the degree of similarity is used as a measure of similarity. Furthermore, at least one of the feature vector related to the shape such as HOG and the feature vector related to the color such as the color histogram is extracted, and the similarity is measured based on at least one of the histogram intersection, the Batacharya coefficient, and the Earth Mover's Distance. The method can be adopted. In addition, a method of measuring similarity by template matching can be adopted. Further, a method of measuring the degree of difference based on at least one of the sum of squares of the differences and the sum of the absolute values of the differences can be adopted instead of the degree of similarity.

Further, in the above embodiment, the tracking process is performed based on the method (called Staple) described in Non-Patent Document 1, but the likelihood map showing the certainty that the tracking object exists in the current frame image. The algorithm for calculating is not limited to the method of the above embodiment. For example, the likelihood map calculation based only on the shape feature, the likelihood map calculation based only on the color feature, and the like may be performed. In addition to calculating the likelihood map by applying a correlation filter, deep learning models such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and RSTM (Long Short-Term Memory) are used. You may go. The present invention selects one of the peaks in consideration of the image similarity such as the center average color when a plurality of peaks appear in the likelihood map, regardless of the likelihood map calculation algorithm. Applicable.

Further, in the above embodiment, the fisheye image is processed without being developed in a plane, but an image in which the fisheye image is developed in a plane may be processed, or an image taken by a normal camera is used as a processing target. May be good.

<Additional notes>
(1) Acquisition means (21) for acquiring the position of the object in the first frame image, and
A tracking means (24) for obtaining the position of the object from the second frame image, which is a frame image after the first frame image, and
An object tracking device (1) comprising the
The tracking means
The feature amount extraction means (25) for extracting the feature amount from the target area of the second frame image, and
With the likelihood calculation means (26), which obtains a likelihood map representing the certainty that the object exists in the target region of the second frame image based on the feature amount.
When there is one peak in the likelihood map, the position of the peak is specified as the position of the object, and when there are a plurality of peaks in the likelihood map, the object in the first frame. The position of the peak selected in consideration of the image similarity representing the similarity between the image region near the position of and the image region near each peak of the second frame is specified as the position of the object. Determining means (27) and
An object tracking device (1), characterized in that.

(2) In the acquisition step (S102) of acquiring the position of the object in the first frame image,
A tracking step of finding the position of the object from the second frame image, which is a frame image after the first frame image, and (S106).
Is an object tracking method that includes
The tracking step
The feature amount extraction step (S302, S304) for extracting the feature amount from the target area of the second frame image, and
With respect to the target region of the second frame image, a likelihood calculation step (S303, S305, S306) for obtaining a likelihood map representing the certainty that the object exists based on the feature amount.
When there is one peak in the likelihood map, the position of the peak is specified as the position of the object, and when there are a plurality of peaks in the likelihood map, the object in the first frame. The position of the peak selected in consideration of the image similarity representing the similarity between the image region near the position of and the image region near each peak of the second frame is specified as the position of the object. The decision step (S307) and
A method for tracking an object, including.

1: Person tracking device 2: Surveillance system 10: Fisheye camera 11: Tracking area 12: Ceiling 13: Person

Claims (9)

An acquisition means for acquiring the position of the object in the first frame image, and
A tracking means for obtaining the position of the object from the second frame image, which is a frame image after the first frame image, and
An object tracking device equipped with
The tracking means
A feature amount extraction means for extracting a feature amount from the target area of the second frame image, and
A likelihood calculating means for obtaining a likelihood map representing the certainty that the object exists in the target region of the second frame image based on the feature amount.
When there is one peak in the likelihood map, the position of the peak is specified as the position of the object, and when there are a plurality of peaks in the likelihood map, the target in the first frame image. The position of the peak selected in consideration of the image similarity representing the similarity between the image area near the position of the object and the image area near each peak of the second frame image is specified as the position of the object. , Positioning means,
An object tracking device comprising. When there are a plurality of peaks in the likelihood map, the position determining means uses the position of the peak having the maximum image similarity among the peaks having a likelihood value equal to or higher than the threshold value as the position of the object. Identify,
The object tracking device according to claim 1, wherein the object tracking device is characterized in that. The threshold is determined for each peak according to the image similarity.
The object tracking device according to claim 2, wherein the object tracking device is characterized by this. The feature amount extracting means extracts a first feature amount, which is a feature amount related to shape, and a second feature amount, which is a feature amount related to color or brightness.
The likelihood calculating means obtains a composite likelihood map obtained by synthesizing a first likelihood based on the first feature amount and a second likelihood based on the second feature amount as the likelihood map.
The object tracking device according to any one of claims 1 to 3, wherein the object tracking device is characterized. The first feature amount is at least one of a HOG feature amount, an LBP feature amount, a SHIFT feature amount, and a SURF feature amount.
The object tracking device according to claim 4, wherein the second feature amount is at least one of a luminance histogram, a color histogram, and a Color Names feature amount. The image similarity is
Determined or determined based on at least one of the difference in image information, including at least one of the average color, average brightness, and representative color, the absolute value of the difference, and the square of the difference in the image region.
At least one of the histogram intersection, the butterfly coefficient, and the Earth Mover's Distance, which is at least one of the first feature amount which is a feature amount related to the shape or the second feature amount which is a feature amount related to color or brightness in the image region. Determined based on, or
Determined by template matching in the image area
The object tracking device according to any one of claims 1 to 5, wherein the object tracking device is characterized. The first frame image and the second frame image are fisheye images obtained by a fisheye camera.
The object tracking device according to any one of claims 1 to 6, wherein the object tracking device is characterized. The acquisition step of acquiring the position of the object in the first frame image, and
A tracking step of finding the position of the object from the second frame image, which is a frame image after the first frame image, and
Is an object tracking method that includes
The tracking step
A feature amount extraction step for extracting a feature amount from the target area of the second frame image, and
With respect to the target area of the second frame image, a likelihood calculation step of obtaining a likelihood map representing the certainty that the object exists based on the feature amount, and
When there is one peak in the likelihood map, the position of the peak is specified as the position of the object, and when there are a plurality of peaks in the likelihood map, the target in the first frame image. The position of the peak selected in consideration of the image similarity representing the similarity between the image area near the position of the object and the image area near each peak of the second frame image is specified as the position of the object. , Positioning steps and
A method for tracking an object, including. A program for causing a computer to perform each step of the method according to claim 8.

Images