JP7474662B2 - Image tracking device, image tracking system, image tracking method and program - Google Patents

Description

The present disclosure relates to an image tracking device, an image tracking system, an image tracking method, and a program.

Patent document 1 describes an invention that is an image processing device that generates composite image data using multiple viewpoint image data obtained by capturing images from multiple viewpoints, and is characterized by comprising a filter processing means that performs a filter process on the multiple viewpoint image data based on distance information that indicates the distance to the subject, and a generation means that generates composite image data by synthesizing the multiple viewpoint image data that has been subjected to the filter process.

JP 2013-065280 A

One example of a known image-based object tracking technique is template matching. This is a process of searching for areas (partial images) that are similar to a template image of the target object from an input overall image (a frame that makes up a video).

If the camera's image capture cycle is sufficiently fast and the apparent change between adjacent frames is sufficiently small, it is possible to track a target object in the image quickly and with high accuracy by performing template matching while referring to the detection results in the immediately preceding frame.

However, when the appearance of an object changes significantly due to a part of the object being "occluded," conventional object tracking technology has difficulty detecting the object using the detection results from the previous frame, and the object may be lost.

The objective of the present disclosure is to provide an image tracking device, an image tracking system, an image tracking method, and a program that can continue tracking an object on an image even when the appearance of the object changes significantly due to a part of the object being "hidden."

According to one aspect of the present disclosure, an image tracking device includes an image acquisition unit that sequentially acquires images from a photographing device, a target information storage unit that extracts a target object captured in the image and stores target information indicating the characteristics of the target object in the image, a shape estimation unit that estimates the three-dimensional shape of the target object from the target object captured in the image and acquires estimated shape information indicating the three-dimensional shape, and an image tracking unit that performs image tracking using the target information and the estimated shape information.

According to one aspect of the present disclosure, an image tracking method includes the steps of: sequentially acquiring images from a photographing device; extracting a target object shown in the image and storing target information indicating the characteristics of the target object in the image; estimating a three-dimensional shape of the target object from the target object shown in the image and acquiring estimated shape information indicating the three-dimensional shape; and performing image tracking using the target information and the estimated shape information.

According to one aspect of the present disclosure, the program causes a computer to execute the steps of sequentially acquiring images from a photographing device, extracting a target object shown in the image and storing target information indicating the characteristics of the target object in the image, estimating a three-dimensional shape of the target object from the target object shown in the image and acquiring estimated shape information indicating the three-dimensional shape, and performing image tracking using the target information and the estimated shape information.

According to each of the above-mentioned aspects, tracking on the image can be continued even if the appearance of the object changes significantly due to a part of the object being "occluded."

FIG. 1 is a diagram illustrating a configuration of an image tracking device according to at least one embodiment of the present disclosure. FIG. 1 is a diagram illustrating a configuration of an image tracking device according to at least one embodiment of the present disclosure. FIG. 2 is an explanatory diagram of a function of a shape estimation unit according to at least one embodiment of the present disclosure. FIG. 2 is a diagram illustrating a process flow of an image tracking device according to at least one embodiment of the present disclosure. 1 is an explanatory diagram of the actions and effects obtained by an image tracking device according to at least one embodiment of the present disclosure. FIG. 11 is a diagram showing a processing flow of an image tracking device according to another embodiment of the present disclosure. FIG. 1 illustrates a configuration of an image tracking system in accordance with at least one embodiment of the present disclosure.

First Embodiment
An image tracking device according to a first embodiment will be described below with reference to Figures 1 to 5. In this embodiment, a system is considered in which an object (drone) is detected on an image while being tracked, and a laser is continuously irradiated at a specific position on the object to be tracked in order to forcibly stop the object.

(Configuration of Image Tracking Device)
FIG. 1 is a diagram showing the configuration of an image tracking device according to the first embodiment.
FIG. 2 is a diagram showing the functional configuration of the CPU of the image tracking device according to the first embodiment.
As shown in FIG. 1, the verification processing device 1 includes a CPU 10, a memory 11, a display 12, an input device 13, and a storage 14, and is configured as a normal computer.

The memory 11 is a so-called main storage device, and is where the instructions and data for the CPU 10 to operate based on the program are stored.

The display 12 is a display device that displays information in a visible manner, and may be, for example, a liquid crystal display or an organic electroluminescence display.

The input device 13 is an input device that accepts operations by a user of the verification processing device 1, and may be, for example, a general mouse, keyboard, touch sensor, etc.

The communication interface 14 is an interface for communication connection between the image tracking device 1 and an external device. In this embodiment, it is communicatively connected to a camera (imaging device) and a laser irradiation device (described later) (not shown).

Storage 15 is a so-called auxiliary storage device, and may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

The function of the CPU 10 will be described with reference to FIG.
The CPU 10 is a processor that controls the overall operation of the verification processing device 1. As shown in Fig. 2, the CPU 10 according to this embodiment performs the functions of an image acquisition unit 100, a target information storage unit 101, a shape estimation unit 102, an image tracking unit 103, and an irradiation point determination unit 104.

The image acquisition unit 100 sequentially acquires the moving images captured by the camera frame by frame (still image).

The target information storage unit 101 acquires and stores target information from the images acquired by the image acquisition unit 100. Here, target information is information that indicates the visual characteristics (position, size, color, shape, etc.) of a target shown in an image, as measured from that single image.

The shape estimation unit 102 acquires and stores estimated shape information from the image acquired by the image acquisition unit 100. Here, the estimated shape information is the three-dimensional shape of the target object estimated from the image of the target object shown in the image. The function of the shape estimation unit 102 will be described later.

The image tracking unit 103 performs image tracking (object tracking processing (tracking) on an image) using the target information acquired by the target information acquisition unit 101 and the estimated shape information acquired by the shape estimation unit 102.

The irradiation point determination unit 104 uses the target information and estimated shape information to determine the laser irradiation point by the laser irradiation device.

(Functions of the shape estimation unit)
FIG. 3 is an explanatory diagram of the function of the shape estimation unit according to the first embodiment.
As shown in Fig. 3, the function of the shape estimation unit 102 is obtained in advance by performing machine learning using a large number of training image data G. Here, multiple training image data G are created based on a shape model M (CG model) that imitates the three-dimensional shape of a target that is expected to fly (in the example shown in Fig. 3, a four-winged drone), and the shape model M is drawn from various positions and angles. By learning using the training image data G created in this way, the shape estimation unit 102 can estimate the three-dimensional shape of the target reflected in the input image (i.e., the three-dimensional shape corresponding to the shape model M) from the input image.

Note that multiple learning image data G are prepared for different types of targets, such as a four-blade drone as shown in FIG. 3, a six-blade drone, and an airplane-type drone. Then, the three-dimensional shape of each type of target is learned.

(Processing flow of the image tracking device)
FIG. 4 is a diagram showing a processing flow of the image tracking device according to the first embodiment.
The processing flow shown in FIG. 4 is executed for each of the frames successively input from the camera, and is repeatedly executed each time a frame is input.

First, the image acquisition unit 100 acquires one image (frame) from the camera (step S001).

Next, the target information acquisition unit 101 determines whether target information has already been extracted and stored from the previously acquired image (step S002).

If the target information is not stored (step S002; NO), a determination is made as to whether or not a new target object (e.g., a drone) is captured in the input image (target input determination) (step S003).

If the input image does not show a new target object (step S003; NO), processing of the currently input image ends.

On the other hand, if the input image shows a new target object (step S003; YES), the characteristics of the target object measured from the image (position, size, color, shape, etc.) are obtained and stored in memory 11 etc. (step S004).

Next, the shape estimation unit 102 estimates the three-dimensional shape of the target object shown in the input image from the image and stores the estimated shape information (step S005).

If the determination process in step S002 finds that the target information has already been stored (step S002; YES), tracking is performed using the already stored target information and estimated shape information (step S006). Here, the image tracking unit 103 tracks the target using a conventional object tracking technique (e.g., template matching using the target information). The image tracking unit 103 also obtains the position and orientation of the target by fitting the three-dimensional shape indicated in the estimated shape information to the target shown in the image.

Next, the image tracking unit 103 determines whether the target has disappeared (i.e., whether tracking was successful) as a result of the tracking process in step S006 (step S007).

If the target is lost during the tracking process in step S007 (step S007; YES), the stored target information and estimated shape information are deleted (step S008), and processing for this frame is terminated.

If the target has not disappeared (step S007; NO), the image tracking unit 103 displays a marker or the like on the image and outputs the target's position, orientation, etc. (step S009).

The irradiation point determination unit 104 determines the laser irradiation point for the target based on the position and attitude of the target obtained in step S009 (step S010). Here, the irradiation point determination unit 104 can determine the shape and current attitude of the approaching target from the estimated shape information. The irradiation point determination unit 104 refers to the shape, position, and attitude of the target to identify a specific position (e.g., the center position of the drone) that is most likely to have the effect of laser irradiation on the target (forced stopping of the drone), and determines that specific position as the irradiation point.

(Action, Effect)
FIG. 5 is an explanatory diagram of the actions and effects obtained by the image tracking device according to the first embodiment.
As described above, the image tracking device 1 of the first embodiment is characterized in that it has, in particular, a shape estimation unit 102 that acquires estimated shape information of a target from an input image, and an image tracking unit 103 that performs image tracking using the target information and the estimated shape information.
The actions and effects obtained by having such characteristics will be described in detail with reference to FIG.

FIG. 5 shows a frame F1 acquired at a certain time and a frame F2 acquired next thereafter.
Frame F1 shows an image of the target before laser irradiation, and is at a stage where it can be tracked based on target information using conventional object tracking technology.

Assume that immediately after frame F1 is acquired, a laser is irradiated onto the target. In this case, as shown in FIG. 5, light is emitted and smoke is generated in the next acquired frame F2 due to the laser irradiation.

Here, in conventional object tracking technology (e.g., template matching), visual features (target information) extracted from an image are used as a template to track the object in the next frame. However, as in the example shown in Figure 5, if part of the target is "hidden" by emitting light or smoke and its appearance changes significantly between frames, conventional object tracking technology has difficulty matching based on the target information and ends up losing sight of the target. Therefore, if tracking is attempted using only the target information, the moment a laser is irradiated onto the object being tracked in the image, tracking is interrupted, and it is not possible to continue irradiating the object with the laser.

However, according to this embodiment, the irradiation point determination unit 104 performs tracking based on estimated shape information in addition to target information. That is, the irradiation point determination unit 104 acquires the position, shape, posture, and the like of the target object shown in the image by fitting the estimated shape information (three-dimensional shape) obtained by the shape estimation unit 102 to the input image. In this case, even if part of the target object is hidden in the image, as in frame F2, the image tracking unit 103 attempts to fit the estimated shape information as a whole based on the visible part. This allows the image tracking unit 103 to continue tracking even if the appearance changes significantly between frames.

In this way, the image tracking device 1 according to this embodiment performs object tracking based on fitting using estimated shape information, making it possible to achieve robust object tracking even in the event of momentary "occlusion."

As a result, it is possible to continue tracking an object on the image even if the object's appearance changes significantly due to part of it being "occluded."

This allows the laser to be continuously aimed at an approaching object (target), more effectively stopping the object's movement.

Second Embodiment
The image tracking device according to the second embodiment will be described below with reference to FIG.

(Processing flow of the image tracking device)
FIG. 6 is a diagram showing a processing flow of the image tracking device according to the second embodiment.
The process flow executed by the image tracking device 1 according to the second embodiment differs from the process flow of the first embodiment (FIG. 4) in that steps S100, S101, and S102 are added. The processes of steps S100, S101, and S102 will be described in detail below.

As shown in FIG. 6, after processing step S002; YES, the image tracking device 1 according to the second embodiment determines whether or not laser irradiation by the laser irradiation device is currently being performed (step S100). Here, the image tracking device 1 may perform the determination of step S002 by receiving a signal indicating whether or not laser irradiation is currently being performed from the laser irradiation device.

If laser irradiation is not being performed (step S100; NO), the shape estimation unit 102 continues the shape estimation process (similar to step S005) on the newly input image (step S101).

Next, the image tracking unit 103 performs tracking processing based only on the target information (i.e., without using shape estimation information) (step S102).

On the other hand, if laser irradiation is in progress (step S100; YES), similar to the first embodiment, the image tracking unit 103 performs tracking processing based on the target information and shape estimation information (step S006).

(Action, Effect)
In other words, the image tracking unit 103 according to the second embodiment performs image tracking using only the target information when the target is not irradiated with a laser, and performs image tracking using the target information and estimated shape information when the target is irradiated with a laser.

When laser irradiation is not performed, it is expected that there will be no significant changes between previous and subsequent frames, so tracking processing is performed using conventional object tracking technology using only target information. This reduces the computational load compared to when tracking processing is always performed using both target information and estimated shape information.

In the second embodiment, when laser irradiation is not in progress (step S100; NO), the shape estimation process (step S101) is repeated. In this manner, the shape estimation unit 102 can perform shape estimation based on multiple input images, thereby improving the accuracy of the shape estimation. This can also improve the accuracy of the laser irradiation performed in step S009.

<Modification>
FIG. 7 is a diagram showing the configuration of an image tracking system according to a modified example of the first and second embodiments.

As shown in Figure 7, the laser irradiation system 9 includes an image tracking system 9A including the image tracking device 1 described in the first and second embodiments and a camera 2 (imaging device), and a laser irradiation device 3.
The camera 2 sequentially transmits frames (still images) constituting a moving image to the image tracking device 1 .
The laser irradiation device 3 acquires information indicating a laser irradiation point from the image tracking device 1, and irradiates a laser toward the laser irradiation point.

In this modified example, the image projection system 9A is further characterized in that the camera 2 has a filter 20 and is equipped with an illuminator 4.

The filter 20 of the camera 2 is an optical filter that attenuates light in a desired frequency band. The filter 20 may be a low-pass filter, a high-pass filter, a band-pass filter, an ND filter, etc. As a result, by inputting an image through the filter 20, it is possible to reduce the light emission of the target caused by the laser irradiation and to reduce the amount of obscuration of the target by the light emission.
In this modified example, the filter 20 is preferably configured to be able to transmit light emitted by an illuminator 4, which will be described later.

The illuminator 4 emits light from the camera 2 side toward the target. By setting the frequency of the light emitted by the illuminator 4 to a band that is well reflected by the target, the target can be more clearly recognized by the camera 2 through the filter 20. The illuminator 4 may also emit white light.

In the above-described modified example, the image tracking system 9A has been described as having both the filter 20 attached to the camera 2 and the illuminator 4, but other embodiments are not limited to this aspect.
That is, the image tracking system 9A according to another embodiment may have only one of the filter 20 and the illuminator 4.

In the above-described embodiment, the various processing steps of the image tracking device 1 are stored in the form of a program on a computer-readable recording medium, and the above-described various processing steps are performed by the computer reading and executing this program. In addition, computer-readable recording media refers to magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductor memories, etc. In addition, this computer program may be distributed to a computer via a communication line, and the computer that receives this distribution may execute the program.

The above program may be for realizing some of the functions described above. Furthermore, it may be a so-called differential file (differential program) that can realize the functions described above in combination with a program already recorded in the computer system.

As described above, several embodiments of the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

<Additional Notes>
The verification device, the verification processing method, and the program described in each embodiment can be understood, for example, as follows.

(1) In a first aspect, the image tracking device 1 includes an image acquisition unit 100 that sequentially acquires images from an image capture device (camera), a target information storage unit 101 that extracts a target object shown in the image and stores target information indicating the characteristics of the target object in the image, a shape estimation unit 102 that estimates the three-dimensional shape of the target object from the target object shown in the image and acquires estimated shape information indicating the three-dimensional shape, and an image tracking unit 103 that performs image tracking using the target information and the estimated shape information.

(2) In the second aspect, the image tracking unit 103 described in (1) performs image tracking using only the target information when the target is not irradiated with a laser, and performs image tracking using the target information and the estimated shape information when the target is irradiated with a laser.

(3) In a third aspect, the image tracking device 1 described in (1) or (2) further includes a laser irradiation point determination unit 104 that determines a laser irradiation point using the target information and the estimated shape information.

(4) In a fourth aspect, the image tracking system 9A includes an image tracking device 1 described in any one of (1) to (3) and an image capturing device 2, and the image capturing device 2 includes a filter 20.

(5) In a fifth aspect, the image tracking system 9 includes an image tracking device 1 described in any one of (1) to (3), an image capture device 2, and an illuminator 4.

(6) In a sixth aspect, the image tracking method includes the steps of: sequentially acquiring images from a photographing device; extracting a target object shown in the image and storing target information indicating the characteristics of the target object in the image; estimating a three-dimensional shape of the target object from the target object shown in the image and acquiring estimated shape information indicating the three-dimensional shape; and performing image tracking using the target information and the estimated shape information.

(7) In the seventh aspect, the program causes a computer to execute the steps of sequentially acquiring images from a photographing device, extracting a target object shown in the image and storing target information indicating the characteristics of the target object in the image, estimating a three-dimensional shape of the target object from the target object shown in the image and acquiring estimated shape information indicating the three-dimensional shape, and performing image tracking using the target information and the estimated shape information.

1 Image tracking device 10 CPU
11 Memory 12 Display 13 Input device 14 Storage 100 Image acquisition unit 101 Target information storage unit 102 Shape estimation unit 103 Image tracking unit 104 Irradiation point determination unit 2 Camera 20 Filter 3 Laser irradiation device 4 Illuminator 9A Image tracking system 9 Laser irradiation system

Claims (6)

an image acquisition unit that sequentially acquires images from an imaging device;
a target information storage unit that extracts a target object shown on the image and stores target information indicating characteristics of the target object on the image;
a shape estimation unit that estimates a three-dimensional shape of a target object from the target object shown on the image and acquires estimated shape information indicating the three-dimensional shape;
an image tracking unit that performs image tracking using the target information and the estimated shape information;
Equipped with
the image tracking unit performs image tracking using only the target information when the target is not irradiated with a laser, and performs image tracking using the target information and the estimated shape information when the target is irradiated with a laser.
Image tracking device. A laser irradiation point determination unit that determines a laser irradiation point using the target information and the estimated shape information,
The image tracking device of claim 1 . An image tracking device according to claim 1 or 2 ;
The imaging device,
The imaging device includes a filter.
Image tracking system. An image tracking device according to claim 1 or 2 ;
The imaging device;
The illuminator and
An image tracking system comprising: acquiring sequential images from an imaging device;
A step of extracting a target object shown on the image and storing target information indicating characteristics of the target object on the image;
a step of estimating a three-dimensional shape of a target object from the target object shown on the image and acquiring estimated shape information indicating the three-dimensional shape;
performing image tracking using the target information and the estimated shape information;
having
in the step of performing image tracking, when the target is not irradiated with a laser, image tracking is performed using only the target information, and when the target is irradiated with a laser, image tracking is performed using the target information and the estimated shape information.
Image tracking methods. On the computer,
acquiring sequential images from an imaging device;
A step of extracting a target object shown on the image and storing target information indicating characteristics of the target object on the image;
a step of estimating a three-dimensional shape of a target object from the target object shown on the image and acquiring estimated shape information indicating the three-dimensional shape;
performing image tracking using the target information and the estimated shape information;
A program for executing
in the step of performing image tracking, when the target is not irradiated with a laser, image tracking is performed using only the target information, and when the target is irradiated with a laser, image tracking is performed using the target information and the estimated shape information.
program .

Images