JP2022050430A - Tracking system, tracking method and tracking program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a tracking system capable of enhancing the tracking precision when the same target is tracked by plural cameras.

SOLUTION: A tracking system for tracking the same target with plural cameras comprises a first camera 100 for capturing a first picture, a second camera 200 for capturing a second picture which is captured at substantially the same angle of view as the first camera 100, and a processor 80. The processor 80 performs first tracking processing for tracking a target on the first picture, second tracking processing for tracking the target on the second picture, reliability degree calculation processing for calculating a first reliability degree as the reliability degree of a target obtained by the first tracking processing, and a second reliability degree as the reliability of the target obtained by the second tracking processing, and control processing for comparing the first reliability degree and the second reliability degree with each other to specify tracking processing having a higher reliability degree, and controlling one tracking processing based on the specified other tracking processing result.

SELECTED DRAWING: Figure 6

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a tracking system, a tracking method and a tracking program for tracking the same object with a plurality of cameras.

Various methods have been proposed for tracking an arbitrary object in the input image sequence or video when the user specifies the object.

Patent Document 1 describes a moving object detection device that detects a moving object from a moving image. The apparatus described in Patent Document 1 removes a fake optical flow using the optical flow of a moving object in the current time and past image frames.

Also known is a method of tracking an object using a plurality of cameras. Patent Document 2 describes a monitoring system that tracks and monitors a moving object by linking a plurality of cameras. In the surveillance system described in Patent Document 2, a plurality of surveillance cameras pass feature information of a tracking target between each other, and each surveillance camera tracks the tracking target using the feature information.

Patent Document 3 describes an image processing device that processes a visible light image to detect and track an object that has entered the target area. A visible light camera and a far-infrared camera are connected to the image processing device described in Patent Document 3, and both cameras capture an image of a target area for detecting an object at substantially the same angle and at substantially the same imaging magnification. Further, the image processing apparatus described in Patent Document 3 generates a far-infrared background subtraction image using a far-infrared background image when there is an image region of a visible light image unsuitable for detecting an object, and images the image. Detects the image that is being used.

In addition, Patent Document 4 describes a target detection device that detects a target object at high speed. The target detection device described in Patent Document 4 includes a detector that detects a far-infrared image and a near-infrared image input through a lens, respectively.

Japanese Patent No. 381657 Japanese Patent Application Laid-Open No. 2003-324720 Japanese Unexamined Patent Publication No. 2014-157452 Japanese Unexamined Patent Publication No. 6-174828

When tracking an object using a single camera as in the device described in Patent Document 1, there is an environment in which tracking cannot be performed properly depending on the characteristics of the camera. For example, when tracking using a single visible camera, the color of the object can be said to be an effective feature for tracking. However, when the tracking environment is an environment where color differences are difficult to find, such as in the dark, it is difficult to track an object using a single visible camera. Further, for example, when tracking using a far-infrared single camera, characteristics can be obtained depending on the temperature day and night, but when other objects of the same temperature are present, it is difficult to track the object.

Further, the surveillance system described in Patent Document 2 transmits the extracted feature information to another surveillance camera device when one surveillance camera device loses track of the tracking target, and the other surveillance camera device transmits the transmitted feature. When the information matches, the video transmission is started. However, since the positions and angles of view taken by each surveillance camera device are usually different, it cannot be said that the feature information received from other surveillance camera devices can be accurately used for subsequent tracking.

On the other hand, it is possible to improve the reliability of tracking by tracking an object using a plurality of cameras as in the image processing apparatus described in Patent Document 3. However, when tracking by using a plurality of cameras at the same time as in stereo vision, there are problems of difference in angle of view and deviation of appearance due to parallax. Ideally, if the angle of view is the same, the position of the object does not change between the cameras, so processing is very easy. However, since the angle of view differs depending on the size of the camera sensor, the focal length, the lens distortion, and the like, it is difficult to eliminate the difference in the angle of view due to the use of a plurality of cameras.

Further, even if the angles of view are adjusted to be substantially the same by using cameras of the same product number, parallax cannot be eliminated in principle as long as a plurality of cameras are used. If image processing is performed on the assumption that the angle of view is almost the same without considering the deviation of the appearance due to parallax, the tracking target cannot be accurately grasped because the assumption is unreasonable, resulting in a decrease in tracking accuracy. There is a problem that it will be gone.

Therefore, it is an object of the present invention to provide a tracking system, a tracking method, and a tracking program capable of improving the tracking accuracy when tracking the same object with a plurality of cameras.

The tracking system according to the present invention is a tracking system that tracks the same object with a plurality of cameras, and is photographed at an angle that is different from that of the first camera that captures the first image and the first camera. A second tracking process for tracking an object on the first image captured by the first camera, which comprises a second camera for capturing the second image and a processing unit. The second tracking process of tracking the object on the second image captured by the second camera, and the object reflected in the image of the specific frame captured by each of the first camera and the second camera. It is characterized by executing a control process for controlling a tracking process for tracking an object on an image captured by one of the cameras after the time when a specific frame is imaged based on the position of.

Another tracking system according to the present invention is a tracking system that tracks the same object with a plurality of cameras, and has a first camera that captures a first image and an angle of view that is different from that of the first camera. A first tracking process that includes a second camera that captures a second image to be captured and a processing unit, and the processing unit tracks an object on the first image captured by the first camera. Based on the second tracking process of tracking the object on the second image captured by the second camera, and the position of the object obtained from the first tracking process and the second tracking process. , One of the features is to control the tracking process and execute the control process of setting the search range of the object.

The tracking method according to the present invention includes a first tracking process for tracking an object on a first image captured by the first camera, and a second camera for capturing images at an angle of view different from that of the first camera. Based on the second tracking process of tracking the object on the second image captured in, and the position of the object in the image of the specific frame captured by each of the first and second cameras. It is characterized by executing a control process for controlling a tracking process for tracking an object on an image captured by one of the cameras after the time when a specific frame is imaged. In addition, other tracking methods according to the present invention include a first tracking process for tracking an object on a first image captured by the first camera, and an angle of view having a parallax with the first camera. One based on the second tracking process of tracking the object on the second image captured by the second camera, and the position of the object obtained from the first tracking process and the second tracking process. It is characterized in that it controls the tracking process of the camera and executes the control process of setting the search range of the object.

The tracking program according to the present invention has a first tracking process for tracking an object on a first image captured by a first camera, and a second image with an image angle different from that of the first camera. A second tracking process for tracking an object on a second image captured by the first camera, and an object appearing in an image of a specific frame captured by each of the first camera and the second camera. It is characterized in that a control process for controlling a tracking process for tracking an object on an image captured by one camera after the time when a specific frame is imaged is executed based on a position. In addition, another tracking program according to the present invention has a first tracking process for tracking an object on a first image captured by a first camera, and an angle of view different from that of the first camera. Based on the second tracking process of tracking the object on the second image captured by the second camera to be imaged, and the position of the object obtained from the first tracking process and the second tracking process. Therefore, it is characterized in that one of the tracking processes is controlled and the control process for setting the search range of the object is executed.

According to the present invention, it is possible to improve the tracking accuracy when tracking the same object with a plurality of cameras.

It is a block diagram which shows one Embodiment of the tracking system by this invention. It is explanatory drawing which shows the example which displayed two images side by side. It is explanatory drawing which shows the example which displayed the tracking result. It is explanatory drawing which shows the operation example of the tracking system. It is a block diagram which shows the modification of the tracking system by this invention. It is a block diagram which shows the outline of the tracking system by this invention. It is a block diagram which shows the other outline of the tracking system by this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present invention, the object means any person or object to be tracked.

FIG. 1 is a block diagram showing an embodiment of a tracking system according to the present invention. The tracking system of the present embodiment tracks the same object with a plurality of cameras, and includes a first camera 10, a second camera 20, a processing unit 30, an output unit 40, and a storage unit 50. And have.

In the following description, the case where the tracking system includes two cameras (first camera 10 and second camera 20) is illustrated, but the number of cameras included in the tracking system is not limited to two. There may be three or more.

In this embodiment, the first camera 10 and the second camera 20 are used to track the same object. Specifically, one camera captures an image having a range of substantially the same angle of view as the other camera. In the following description, the image captured by the first camera is referred to as a first image, and the image captured by the second camera is referred to as a second image. Further, the second image is shot at substantially the same angle of view as the first image.

The positional relationship between the first camera 10 and the second camera 20 may be set so that the target object can be tracked by, for example, stereo vision. Here, the positional relationship means the position, orientation, and magnification of the camera set so that two images are shot at substantially the same angle of view as described above.

In principle, parallax cannot be eliminated due to the characteristic of tracking the same object using a plurality of cameras. Therefore, the angle of view of each camera (image range, size at a certain distance, etc.) is adjusted to be substantially the same with respect to the angle of view of one camera. This setting is the same even when there are three or more cameras.

Each camera may be set using a device that fixes the positional relationship with each other. Further, the installation location of the first camera 10 and the second camera 20 may not be fixed as long as the mutual positional relationship does not change. For example, these cameras may be attached to the same moving body and moved, and the angle of view from the initial state may change.

In addition, each camera is set so that the object exists within the angle of view at the start of tracking. However, the object does not have to be within the angle of view except at the start of tracking. The first camera 10 and the second camera 20 input the captured video to the processing unit 30.

The content of the camera used in the present embodiment is arbitrary as long as it is a camera capable of photographing an object. Specifically, the first camera 10 and the second camera 20 may be realized by a camera provided with the same type of sensor, or may be realized by a camera provided with different types of sensors. For example, the first camera may be a visible light camera and the second camera may be a far infrared camera. The mode of the first camera 10 and the second camera 20 is not limited to the visible light camera and the far-infrared camera, and may be, for example, a near-infrared camera.

Further, the frame rates of the first camera 10 and the second camera 20 may be the same or different. For example, the first camera 10 may capture a high resolution image at a lower frame rate than the second camera 20, and the second camera 20 may capture a higher frame than the first camera 10. Low resolution images may be taken at a rate.

The storage unit 50 stores information used for various processes by the processing unit 30 described later. The storage unit 50 is realized by, for example, a magnetic disk device or the like.

The output unit 40 displays the video input to the processing unit 30. For example, the output unit 40 may display two images shot at the same time side by side so that they can be compared. The output unit 40 is realized by, for example, a display device or a touch panel device. FIG. 2 is an explanatory diagram showing an example in which two images are displayed side by side. In the example shown in FIG. 2, it is shown that the image taken by the visible light camera is displayed on the left side and the image taken by the far infrared camera is displayed on the right side.

The processing unit 30 inputs an image taken by the first camera 10 (that is, a first image) and an image taken by the second camera 20 (that is, a second image). The processing unit 30 receives from the user the designation of an object on one of the input images. In the present embodiment, the processing unit 30 accepts the designation of the object on the first image.

The object is designated by selecting a range including the object on the first image. The method of specifying the object is arbitrary. For example, when the output unit 40 is realized by a display device, an object may be designated by a range selection operation using a pointing device for the first image displayed on the output unit 40. Further, for example, when the output unit 40 is realized by the touch panel, the object may be designated by the user's range selection operation for the first image displayed on the output unit 40.

When an object on the first image is designated, the processing unit 30 specifies a designated range including the object as a search range. Hereinafter, this search range may be referred to as a first search range. At the same time, the processing unit 30 specifies a range in which the same object exists on the second image as a search range. Hereinafter, this search range may be referred to as a second search range.

Here, the search range is an area where the detection process of the object is performed. Since the detection process is a process of searching for and detecting an object having the same appearance as the learned tracking target, the search range can be said to be a range on the image on which the search is performed. In addition, the search is a candidate for all grids in which an image is divided into predetermined sizes (for example, 5 pixels × 5 pixels each, 10 pixels × 10 pixels each, etc.), and the appearance is the same as the learned tracking target. Is a process of searching from the candidates and using it as a detection result.

As described above, there is a parallax between the first image and the second image. Therefore, the processing unit 30 specifies the second search range in consideration of the parallax (displacement). For example, the processing unit 30 compares the frame one frame before the specified time with the frame at the specified time in the range of the rectangle specified in the first image, and determines the magnitude of the movement of the object and the frame at the specified time. Estimate the orientation. The processing unit 30 may estimate the magnitude and direction of movement using, for example, optical flow. Next, the processing unit 30 identifies a rectangle at the same position as the rectangle specified in the first image on the second image. Similar to the first image, the processing unit 30 compares the rectangle one frame before the designation with the frame at the time of designation for the specified rectangle on the second image, and estimates the magnitude and direction of the movement of the object. do. Then, the processing unit 30 estimates a place where the estimated object has a similar size and direction as a place where the specified object exists on the second image, and specifies the second search range. ..

In this way, since the search range on the other image is specified based on the magnitude and direction of the movement of the object on one image, the object can be specified in consideration of the deviation existing between the two images. Further, by automatically specifying the object on the other image in which the object is not specified, it is possible to reduce the trouble of designating the object. That is, in the present embodiment, the user only has to specify the object on one image, and the object on the other image is automatically estimated. Therefore, despite the tracking technique using a plurality of different types of cameras. It can be said that the time and effort required for input does not increase compared to the case of a single camera. After that, the processing unit 30 determines the search range of the object in parallel from the images captured by each camera, and performs tracking processing.

When the processing unit 30 specifies each search range, the processing unit 30 calculates the reliability of the object at a fixed frame interval for each video. This reliability is a degree of similarity indicating the likeness of an object, and is calculated based on the tracking information from the start of tracking for each video. Here, the tracking information from the start of tracking includes each image showing the object acquired at the time of tracking and information indicating each position of the object (hereinafter, referred to as a template). Templates are created for each video (for each camera that shoots).

The processing unit 30 calculates the reliability of the object based on the degree of similarity in appearance when compared with the template, and the degree of proximity and plausibility of the position of the object estimated according to the movement. At this time, the processing unit 30 may calculate the reliability of the object in each frame while updating the template of the object at any time. Further, the processing unit 30 learns a model of the object (visible model, moving model) based on an image acquired at any time during the search and information indicating the movement of the object, and uses the learned model to learn the object. The reliability may be calculated.

The processing unit 30 may update the template of the object based on the reliability calculated for each video, and the relative reliability (relative) when the reliability calculated for each video is compared. The template of the object may be updated based on the reliability). For example, when the calculated reliability is lower than the reliability calculated in another video, the processing unit 30 may lower (or set) the update weight of the object specified in the image. Further, for example, when the reliability is high and the relative reliability is not high, the processing unit 30 may lower (or set) the update weight of the object in this case as well.

The method is arbitrary as long as the reliability of the object can be calculated based on the information acquired by the tracking process of the object on each video (specifically, the tracking information from the start of tracking). be. That is, the reliability is a degree indicating the likelihood of being tracked, and may be calculated by an equation that can be determined that the higher the value, the higher the similarity.

Next, the processing unit 30 reflects the position and size of the object in one tracking process determined to have high reliability in the setting of the search range of the object in the other tracking process. Specifically, the processing unit 30 compares the reliability calculated from each video, and identifies the video and the object for which the highest reliability is calculated at the same time. Then, the processing unit 30 determines the search range of the video other than the video for which the highest reliability is calculated by using the information indicating the range (position and size) of the specified object. That is, it can be said that the processing unit 30 shares the information of the object for which the highest reliability has been calculated with all the other images. After that, the tracking process is continued within the search range determined for each image.

The processing unit 30 may change the method of determining the search range according to the calculated reliability and the tracking result of the previous frame. Specifically, the processing unit 30 may change the method of determining the search range depending on whether or not the tracking result of the previous frame exists and whether or not the reliability is high or low. For example, when the tracking result of the previous frame does not exist or the reliability is low, the processing unit 30 searches for the tracking target using the entire image as the search range. At this time, the processing unit 30 may specify the object from the entire image by using only a template close to the size of the object specified by the latest tracking result in the past.

In this way, by narrowing down the search range of each camera image and searching for and adopting the candidate with a certain level of reliability or the highest reliability within that range, it is possible to adjust the tracking result while shortening the search time. .. If the camera image on the shared side does not have a certain degree of reliability or there is no matching template, the processing unit 30 may adopt the shared tracking information as it is.

Further, in the present embodiment, since the object is tracked complementarily by a plurality of cameras, parallax occurs in each camera. Taking advantage of this parallax, the processing unit 30 may determine the search range using the depth from the camera to the object.

When determining the search range using depth, first of all, the user actually calibrates the knowledge that the distance to the object will deviate to this extent within a certain depth range in advance. It is calculated and the calculation result is stored in the storage unit 50. That is, the storage unit 50 stores the deviation (difference) according to the distance to the target and the depth range calculated by the calibration. The processing unit 30 calculates the depth to the object based on the deviation observed at the time of actual tracking, estimates the moving range of the object on the image of the current frame from the depth calculated in the image of the previous frame, and searches. Specify the range. By narrowing the search range in this way, it is possible to apply the template only to a part of the range on the image, so that the speed of the tracking process can be improved.

In addition, the storage unit 50 may store the depth range and the search range in association with each other. In this case, the processing unit 30 calculates the depth range of the object based on the deviation observed at the time of actual tracking, and from the calculated depth range to another tracking process based on the correspondence stored in the storage unit 50. The search range to be set may be determined.

After determining the search range, the processing unit 30 performs tracking processing (object detection processing) again, and tracks the object on the image displayed on the output unit 40, for example, by surrounding the object with a rectangle. The results may be displayed at any time. By displaying such a tracking result, it becomes possible to visually track the object.

At this time, the processing unit 30 may display the tracking result on the video with higher reliability in a specific mode. For example, the processing unit 30 displays an object on a highly reliable image so as to be surrounded by a rectangle with a thick dark frame, and encloses an object on the image with low reliability as a rectangle with a thin frame of a light color. It may be displayed (or not enclosed).

FIG. 3 is an explanatory diagram showing an example of displaying the tracking result. In the example shown in FIG. 3, as a result of performing the tracking process on the image illustrated in FIG. 2, the reliability of the tracking result by the image taken by the second camera is higher than that by the image taken by the first camera. Indicates that it was continuously high. By doing so, it becomes possible to visually grasp which video object tracking result is used for the tracking process.

The processing unit 30 is realized by the CPU of a computer that operates according to a program (tracking program). For example, the program may be stored in the storage unit 50, and the CPU may read the program and operate as the processing unit 30 according to the program. Further, the processing unit 30 may be realized by dedicated hardware.

Next, the operation of the tracking system of the present embodiment will be described. FIG. 4 is an explanatory diagram showing an operation example of the tracking system of the present embodiment. First, image adjustment is performed in advance between the image captured by the first camera 10 (that is, the first image) and the image captured by the second camera 20 (that is, the second image). (Step S11). Specifically, both cameras are adjusted so that the first camera 10 and the second camera 20 can shoot an image having substantially the same angle of view.

After the adjustment, the processing unit 30 accepts the user's designation for the object on the first image (step S12). When the object on the first image is designated, the processing unit 30 specifies the search range on the first image (first search range) and the search range on the second image (second). The search range of) is specified (step S13).

When the search range on each image is specified, the processing unit 30 tracks the object on each image (steps S14 and S15), and calculates the reliability of the tracked object at regular frame intervals (steps S14 and S15). Steps S16, S17). The processing unit 30 compares the reliability calculated for each video, and identifies the video and the object for which the highest reliability is calculated (step S18).

The processing unit 30 controls the tracking process of the other based on the identified tracking process result (steps S19 and S20). Specifically, the processing unit 30 determines the search range by adjusting the tracking result, and repeats the processes of steps S14 and S15 and subsequent steps for the image of the next frame.

As described above, in the present embodiment, the first camera 10 captures the first image, and the second camera 20 captures the second image captured at substantially the same angle of view as the first camera. do. Then, the processing unit 30 provided in the tracking system executes a first tracking process for tracking the object on the first image and a second tracking process for tracking the object on the second image. .. Further, the processing unit 30 has the reliability of the object obtained in the first tracking process (first reliability) and the reliability of the object obtained in the second tracking process (second reliability). ) And the reliability calculation process is executed. Then, the processing unit 30 compares the first reliability with the second reliability to identify the tracking process having higher reliability, and based on the identified tracking process result, the other tracking process. Execute the control process that controls the process. Therefore, it is possible to improve the tracking accuracy when tracking the same object with a plurality of cameras.

Further, in the present embodiment, the search range is finely adjusted (corrected) by sharing the tracking result in consideration of the visual deviation (parallax) when a plurality of cameras represented by stereoscopic vision are used. You can track the object more robustly.

Further, in the present embodiment, the reliability of the object is calculated independently for each image, and the tracking information of the object specified from the highly reliable image is reflected in other images, so that the camera used for tracking is used. It doesn't matter if the sensor or frame rate is different. Therefore, it is possible to track the target complementarily with different types of cameras, and it is possible to improve the tracking accuracy. In other words, in the present invention, since tracking is performed based on the most reliable tracking information among a plurality of cameras, it can be said that the characteristically superior scene of each camera is automatically selected and tracked. ..

Specifically, in the present embodiment, an object is tracked in parallel using images of a plurality of different or similar cameras (for example, a far-infrared camera and a visible light camera) having substantially the same angle of view. At the same time, the tracking information on the high reliability side is shared with the low reliability side to correct the tracking information. For example, when a camera including different types of sensors is combined, it becomes possible to track a scene that one type of camera is not good at while complementing it with another type of camera.

Further, in the present embodiment, since it is sufficient to combine existing single cameras and adjust the angles of view of the cameras to be substantially the same, a special lens mechanism as described in Patent Document 4 is not required. In addition, if the positional relationship between the cameras (relationship of camera position, orientation, magnification, etc.) is not changed, the object can be tracked even if the camera is moved during tracking. Therefore, it is also possible to apply the tracking system of the present embodiment to a mobile camera system.

Next, a modification of the present embodiment will be described. In the example shown in FIG. 1, the tracking system includes a processing unit 30, and the image input from the first camera 10 (first image) and the image input from the second camera (second image) by the processing unit 30. Tracking processing and reliability calculation processing were performed for both of the video).

In this modification, a configuration will be described in which processing for images taken by each camera is realized by different means. FIG. 5 is a block diagram showing a modified example of the tracking system according to the present invention. The tracking system exemplified in FIG. 5 includes a first camera 10, a second camera 20, a tracking unit 31, a tracking unit 32, a reliability calculation unit 33, a reliability calculation unit 34, and a control unit 35. And an output unit 40 and a storage unit 50.

Since the contents of the first camera 10, the second camera 20, the output unit 40, and the storage unit 50 are the same as those in the above embodiment, detailed description thereof will be omitted. Further, the contents performed by the tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 are the same as the contents performed by the processing unit 30 of the above-described embodiment.

Specifically, the tracking unit 31 and the reliability calculation unit 33 perform processing on the image (first image) captured by the first camera 10. Further, the tracking unit 32 and the reliability calculation unit 34 perform processing on the image (second image) taken by the second camera 20. The processing performed by the tracking unit 31 and the processing performed by the tracking unit 32, and the processing performed by the reliability calculation unit 33 and the processing performed by the reliability calculation unit 34 are different except that the target video or image is different. Each is the same.

The tracking unit 31 tracks the object on the first image (first tracking process). Further, the tracking unit 32 tracks the object on the second image (second tracking process). The method of tracking the object is the same as the method performed by the processing unit 30 described above.

The reliability calculation unit 33 calculates the reliability (first reliability) of the object obtained in the first tracking process. In addition, the reliability calculation unit 34 calculates the reliability (second reliability) of the object obtained in the second tracking process. The method of calculating the reliability is the same as the method performed by the processing unit 30 described above.

The control unit 35 compares the first reliability with the second reliability to identify the tracking process having higher reliability, and based on the identified tracking process result, performs the other tracking process. Control. Specifically, the control unit 35 reflects the position and size of the object in one tracking process determined to have high reliability in the setting of the search range of the object in the other tracking process. The method of reflecting the setting of the search range is the same as the method performed by the processing unit 30 described above.

The tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 are realized by a computer CPU that operates according to a program (communication information reference program). Further, the tracking unit 31, the tracking unit 32, the reliability calculation unit 33, the reliability calculation unit 34, and the control unit 35 may be realized by dedicated hardware, respectively.

Further, for example, the first camera 10, the tracking unit 31, and the reliability calculation unit 33 are configured as an integrated tracking device, and the second camera 20, the tracking unit 32, and the reliability calculation unit 34 are combined. It may be configured as an integrated tracking device. Then, each tracking device may carry out each process in response to an instruction from the control unit 35.

Even with such a configuration, it is possible to improve the tracking accuracy when tracking the same object with a plurality of cameras.

Next, the outline of the present invention will be described. FIG. 6 is a block diagram showing an outline of the tracking system according to the present invention. The tracking system according to the present invention is a tracking system that tracks the same object (for example, a tracking target) with a plurality of cameras, and is a first camera 100 (for example, a first camera 10) that captures a first image. ), A second camera 200 (for example, the second camera 20) that captures a second image taken at substantially the same angle of view as the first camera 100, and a processing unit 80 (for example, the processing unit 30). ) And.

The processing unit 80 was obtained by the first tracking process of tracking the object on the first image, the second tracking process of tracking the object on the second image, and the first tracking process. A reliability calculation process for calculating the first reliability, which is the reliability of the object, and the second reliability, which is the reliability of the object obtained in the second tracking process, and the first reliability. And the second reliability are compared to identify a tracking process having a higher reliability, and based on the identified tracking process result, a control process for controlling the other tracking process is executed.

With such a configuration, it is possible to improve the tracking accuracy when tracking the same object with a plurality of cameras.

Here, the first camera 100 and the second camera 200 may be realized by cameras provided with different types of sensors. When each camera is realized by a camera equipped with different types of sensors, the content of the image taken by each camera is significantly different. However, in the present invention, the reliability of each captured image is calculated, and the tracking process of other images is controlled based on the tracking process result (for example, the search range) of the image for which the higher reliability is calculated. .. Therefore, it is possible to track a scene that one type of camera is not good at while complementing it with another type of camera.

Specifically, the first camera 100 may be a visible light camera, and the second camera 200 may be a far infrared camera. In this case, the target can be tracked in a particularly complementary manner, and tracking that makes the best use of the characteristics of each camera becomes possible.

Further, the first camera 100 captures a high-resolution image at a lower frame rate than the second camera 200, and the second camera 200 has a lower resolution at a higher frame rate than the first camera 100. You may take an image of.

Further, the processing unit 80 may reflect the position and size of the object in one tracking process determined to have high reliability in the setting of the search range of the object in the other tracking process. That is, since the search range of other tracking processes is determined from the tracking result of the object with high reliability, tracking can be performed while correcting the search range even when a plurality of cameras whose angles of view do not completely match are used. Therefore, it becomes possible to accurately track the same object. That is, in the present invention, it is set so that the search range of the other image can be absorbed by reflecting the tracking processing result of one of them against the deviation of the appearance on the image that occurs when a plurality of cameras are used. I can say.

Further, the tracking system may include a storage unit (for example, a storage unit 50) that stores the depth range and the search range in association with each other. Then, the processing unit 80 may refer to the storage unit and specify the search range from the depth range of the object calculated based on the difference observed at the time of tracking.

Further, the first camera 100 and the second camera 200 may be attached to the same moving body to track the same object. In the present invention, if both cameras are installed so that they can shoot at substantially the same angle of view, it is not necessary to match the angles of view exactly. Can be improved.

Further, the processing unit 80 may update the template information of the object used for calculating the reliability at regular frame intervals, and calculate the reliability of each frame using the updated template information.

FIG. 7 is a block diagram showing another outline of the tracking system according to the present invention. The tracking system illustrated in FIG. 7 is a tracking system that tracks the same object with a plurality of cameras, and has substantially the same image as the first camera 100 that captures the first image and the first camera 100. A second camera 200 that captures a second image taken at a corner, a first tracking unit 81 that tracks an object on the first image (for example, a tracking unit 31 and a processing unit 30), and a first The first reliability, which is the reliability of the object obtained by the second tracking unit 82 (for example, the tracking unit 32, the processing unit 30) for tracking the object on the image of the second image and the first tracking unit 81. The second reliability, which is the reliability of the object obtained by the first reliability calculation unit 83 (for example, the reliability calculation unit 33, the processing unit 30) for calculating the degree and the second tracking unit 82, is calculated. The second reliability calculation unit 84 (for example, the reliability calculation unit 34, the processing unit 30) to be calculated is compared with the first reliability and the second reliability, and the tracking process with higher reliability is performed. It includes a control unit 85 (for example, a control unit 35 and a processing unit 30) that controls the other tracking process based on the identified and identified tracking process result.

Such a configuration can also improve the tracking accuracy when tracking the same object with a plurality of cameras.

10 1st camera 20 2nd camera 30 Processing unit 31, 32 Tracking unit 33, 34 Reliability calculation unit 35 Control unit 40 Output unit 50 Storage unit

The tracking system according to the present invention is a tracking system that tracks the same object with a plurality of cameras, and includes a processing unit that inputs images from at least two or more visible light cameras, and the processing unit is the first visible. The first detection process for detecting an object on the first image captured by the optical camera, and the second visible light camera for capturing the image at an angle different from that of the first visible light camera. Based on the second detection process for detecting an object on the second image and the position of the object in the image of a specific frame captured by each of the first visible light camera and the second visible light camera. , A control process that controls a tracking process that tracks an object on an image captured by at least one of a first visible light camera and a second visible light camera after the time when a specific frame is imaged. It is characterized by doing.

The tracking method according to the present invention includes a first detection process for detecting an object on a first image captured by a first visible light camera, and imaging with an angle of view different from that of the first visible light camera. A second detection process for detecting an object on the second image captured by the second visible light camera, and a specific frame captured by each of the first visible light camera and the second visible light camera. Tracks an object on an image captured by at least one of a first visible light camera and a second visible light camera after the time when a particular frame was imaged, based on the position of the object in the image. It is characterized by executing a control process for controlling the tracking process .

The tracking program according to the present invention has a first detection process for detecting an object on a first image captured by a first visible light camera, and an angle of view different from that of the first visible light camera. The second detection process for detecting an object on the second image captured by the second visible light camera to be imaged, and the images taken by the first visible light camera and the second visible light camera, respectively. An object on an image captured by at least one of a first visible light camera and a second visible light camera after the time when the specific frame was imaged, based on the position of the object in the image of the specific frame. It is characterized in that a control process for controlling a tracking process for tracking an object is executed .

Claims (13)

A tracking system that tracks the same object with multiple cameras.
The first camera that captures the first image and
A second camera that captures a second image taken at substantially the same angle of view as the first camera, and a second camera.
Equipped with a processing unit
The processing unit
The first tracking process for tracking the object on the first image, and
A second tracking process for tracking the object on the second image,
The first reliability, which is the reliability of the object obtained in the first tracking process, and the second reliability, which is the reliability of the object obtained in the second tracking process, are obtained. The reliability calculation process to be calculated and
A control that compares the first reliability with the second reliability to identify a more reliable tracking process and controls the other tracking process based on the identified tracking process result. A tracking system characterized by performing processing and performing. The tracking system according to claim 1, wherein the first camera and the second camera are realized by a camera provided with different types of sensors. The tracking system according to claim 1 or 2, wherein the first camera is a visible light camera and the second camera is a far infrared camera. The first camera captures high resolution video at a lower frame rate than the second camera.
The tracking system according to any one of claims 1 to 3, wherein the second camera captures a low-resolution image at a higher frame rate than the first camera. Of claims 1 to 4, the processing unit reflects the position and size of the object in one tracking process determined to be highly reliable in the setting of the search range of the object in the other tracking process. The tracking system according to any one of the above. Equipped with a storage unit that stores the depth range and the search range in association with each other.
The processing unit refers to the storage unit, and according to any one of claims 1 to 5, the search range is specified from the depth range of the object calculated based on the difference observed at the time of tracking. Tracking system. The tracking system according to any one of claims 1 to 6, wherein the first camera and the second camera are attached to the same moving object and track the same object. The processing unit updates the template information of the object used for calculating the reliability at regular frame intervals, and calculates the reliability of each frame using the updated template information. The tracking system according to any one of the following items. A tracking system that tracks the same object with multiple cameras.
The first camera that captures the first image and
A second camera that captures a second image taken at substantially the same angle of view as the first camera, and a second camera.
A first tracking unit that tracks the object on the first image,
A second tracking unit that tracks the object on the second image,
A first reliability calculation unit that calculates a first reliability, which is the reliability of the object obtained by the first tracking unit, and a first reliability calculation unit.
A second reliability calculation unit that calculates a second reliability, which is the reliability of the object obtained by the second tracking unit, and a second reliability calculation unit.
A control that compares the first reliability with the second reliability to identify a more reliable tracking process and controls the other tracking process based on the identified tracking process result. A tracking system characterized by having a department. A tracking method that tracks the same object with multiple cameras.
The first camera captures the first image,
The second camera captures a second image taken at substantially the same angle of view as the first camera.
The processing unit performs a first tracking process for tracking the object on the first image and a second tracking process for tracking the object on the second image.
The processing unit has a first reliability, which is the reliability of the object obtained in the first tracking process, and a second reliability, which is the reliability of the object obtained in the second tracking process. Performs reliability calculation processing to calculate the reliability of
The processing unit compares the first reliability with the second reliability to identify a tracking process having a higher reliability, and based on the identified tracking process result, the other tracking process is performed. A tracking method characterized by performing a control process that controls the process. The tracking method according to claim 10, wherein a second camera having a sensor different from the sensor used by the first camera for capturing an image captures the second image. A tracking program that applies to computers that track the same object with multiple cameras.
To the computer
The first tracking process of tracking the object on the first image captured by the first camera,
A second tracking process for tracking the object on the second image captured by the second camera, which takes a picture at substantially the same angle of view as the first camera.
The first reliability, which is the reliability of the object obtained in the first tracking process, and the second reliability, which is the reliability of the object obtained in the second tracking process, are obtained. Reliability calculation processing to calculate, and
A control that compares the first reliability with the second reliability to identify a more reliable tracking process and controls the other tracking process based on the identified tracking process result. A tracking program to perform the process. The tracking program according to claim 12, wherein the sensor of the camera used for imaging is different between the first image and the second image.

Images