JP2009205611A - Image target detection tracking device, image target detection tracking method, and image target detection tracking program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect a specific target from an image and to track it. <P>SOLUTION: A target candidate detecting means 1d reads a template including a target object from a template image storage means 1c, calculates correlation between the template and a division image obtained by dividing an input image by a prescribed unit, and selects a target candidate on the basis of a result of calculation. Also, a candidate tracking means 1f searches for a target candidate to be tracked included in the most recent input image. When the target candidate to be tracked is searched for, tracking of the target candidate is continued. Meanwhile, when the target candidate to be tracked is not searched for, the target candidate to be tracked is deleted from the target candidate. In this way, only the object stably tracked is left as a target candidate. A target specifying means 1g specifies the target candidate to be stably tracked and selected as a target candidate as a target. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image target detection and tracking apparatus, an image target detection and tracking method, and an image target detection and tracking program, and more particularly to an image target detection and tracking apparatus, an image target detection and tracking method, and an image target detection and tracking program for detecting and tracking a target from an input image. About.

  2. Description of the Related Art Conventionally, practical application of an image target detection / tracking apparatus that detects and tracks a target of an object using an image captured by an infrared camera or the like has been promoted in various fields. In fields where ships are subject to detection, such as ensuring the safety of maritime traffic and rescue of marine disasters, application to automation of detection and tracking of ships navigating at sea is being promoted.

By the way, detection of a target using an image sensor is not necessarily accurate at present, but has an advantage that it can be monitored with a relatively simple device. Therefore, even in the field of detecting ships, ships using image sensors that detect and tag ships by inter-frame difference method and track using feature quantities such as motion characteristics of the tagged ships. A technique for automatically detecting and tracking the above has been proposed (see, for example, Non-Patent Document 1).
Yada, “Ship Tracking with Image Sensor”, 4th Electronic Navigation Research Institute Research Presentation, June 2004

  By the way, at the site of marine rescue, it is required to quickly discover and carry out activities such as lifeboats that float on the sea. For this reason, he searches extensively on the sea using helicopters and other aircraft. At this time, the sea is photographed with an infrared camera that can be used at night, and the target is detected using the image. However, floating objects such as lifeboats on the sea are small, and there is a problem that detection is not easy compared to an object having a predetermined size such as a ship.

  In particular, in an image taken from a distance such as an image taken from an aircraft, an image of a lifeboat or a floating object floating on the sea is very small, and the size of the surrounding wave front is not different. For this reason, it is very difficult to identify the target suspended matter from the image by distinguishing it from the surrounding wavefront, and the practical application of automatic detection has not progressed. However, visual target detection and tracking is a very labor intensive task, and automation of target detection and tracking is desired.

  The same problem occurs when detecting a target from an image in which an object having a shape similar to a detection target appears temporarily, not limited to a marine rescue site. For this reason, there is a need for a technique that excludes an object that temporarily appears from such an image from a target and reliably detects and tracks a specific target.

  The present invention has been made in view of the above points, and an image target detection and tracking apparatus, an image target detection and tracking method, and an image target capable of detecting and tracking a specific target from captured images. An object is to provide a detection tracking program.

  In order to solve the above problems, in an image target detection and tracking device that detects and tracks a target from an input image, the template is read from a template storage unit that stores a template image including the target object, and the template and the input image are read. Target candidate detection means for calculating a correlation with a divided image divided in a predetermined unit and registering an object included in the divided image whose calculation result exceeds a predetermined reference value as a target candidate, and a target candidate for each target candidate The target candidate of the tracking target is searched for in the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate. If the candidate can be searched, the target candidate of the tracking target is continuously tracked. If the target candidate of the tracking target cannot be searched, the target candidate of the tracking target is continued. Candidate tracking means for deleting from the target candidates, and when the target candidates are continuously tracked by the candidate tracking means for a predetermined period and are detected as target candidates by the target candidate detecting means, There is provided an image target detection and tracking device characterized by having a target specifying means for specifying as a target. In addition, an image target detection and tracking method is provided in which a processing procedure is executed by the target candidate detection means, candidate tracking means, and target specifying means. In order to solve the above problems, an image target detection / tracking program is provided that causes a computer to function in the same manner as the image target detection / tracking apparatus.

  In the disclosed image target detection and tracking apparatus, image target detection and tracking method, and image target detection and tracking program, when an input image is captured, the target candidate detection means and candidate tracking means are activated. The template including the target object is read from the template storage unit by the target candidate detection unit, and the correlation between the divided image obtained by dividing the input image by a predetermined unit and the template is calculated. Then, an object included in a divided image whose calculation result exceeds a predetermined reference value is registered as a target candidate. Further, the candidate tracking unit performs tracking processing for each target candidate with the target candidate as a tracking target, and searches for target candidates for tracking included in the latest input image. When the target candidate to be tracked is searched, the tracking of the target candidate is continued. On the other hand, when the target candidate for tracking is not searched, the target candidate for tracking is deleted from the target candidates. Thus, only those continuously detected remain as target candidates. Then, a target candidate that can be stably tracked for a predetermined period and is selected as a target candidate is specified as a target by the target specifying means.

  According to the disclosed image target detection and tracking apparatus, image target detection and tracking method, and image target detection and tracking program, target candidates that satisfy a predetermined criterion are selected from the input image, and tracking of each target candidate is started. Then, after the tracking is started, the tracking can be stably performed, and the candidate detected as the target candidate is specified as the target. Thereby, an object having a shape similar to the detection target that appears temporarily in the image can be excluded from the target candidates, and a specific target can be reliably detected and tracked.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram of an image target detection and tracking apparatus according to an embodiment.
The image target detection / tracking apparatus 1 includes an image input unit 1a, an image storage unit 1b, a template image storage unit 1c, a target candidate detection unit 1d, a candidate template image storage unit 1e, a candidate tracking unit 1f, and a target specifying unit 1g.

  The image input means 1a acquires image data photographed by the camera 2 and stores it in the image storage means 1b. The captured image data is sequentially transmitted from the camera 2. When the image input means 1a receives image data from the camera 2, the image input means 1a updates the image data stored in the image storage means 1b.

The image storage unit 1b stores the latest image data acquired from the camera 2 by the image input unit 1a.
The template image storage unit 1c stores an image of the target template including the target object.

  The target candidate detection unit 1d is activated at a predetermined timing, for example, when an input image is acquired, and detects a target candidate using template matching. First, the target template stored in the template image storage unit 1c is read out. Also, the latest image data stored in the image storage means 1b is read out and divided into areas of a predetermined unit. Then, a correlation between the divided image divided into predetermined units and the target template is calculated, and a divided image whose calculation result exceeds a predetermined reference value determined in advance is obtained. An object included in this divided image is set as a target candidate. The area in which the target candidate is detected is compared with the area in which the already registered target candidate is detected by the candidate tracking unit 1f. It is determined that it is a registered target candidate. If there is no matching registered target candidate, the detected target candidate is newly registered. Then, the divided image including the detected target candidate is stored in the candidate template image storage unit 1e as a tracking candidate template. Further, an identification symbol (hereinafter referred to as ID) is assigned to the target candidate, and an area on the input image where the target candidate is detected is stored in association with the ID. Note that the correlation of images can be obtained by various methods such as a method of calculating a sum of absolute values of differences or a sum of squares of differences. For calculating the correlation of images, it is assumed that such a known method is appropriately selected and used, and details thereof are omitted.

In the candidate template image storage unit 1e, candidate images registered as target candidates by the target candidate detection unit 1d are stored as candidate templates.
The candidate tracking unit 1f is activated at a predetermined timing such as when an input image is acquired, and performs tracking processing for each target candidate using the target matching as a tracking target using template matching. The tracking process is performed using an input image in which a target candidate is first detected or an image taken after the input image on which the previous tracking process has been performed. First, the candidate template of the target candidate is read out from the candidate template image storage unit 1e, and the latest image data stored in the image storage unit 1b is read out and divided into predetermined regions to be matched with the candidate template. At this time, the matching is centered on an area on the input image where the target candidate stored in association with the target candidate is detected or an area on the input image where the target candidate to be tracked last time is detected. This is performed in a predetermined area. When a target candidate to be tracked is detected, the position of the detected area is stored in association with the target candidate ID, and the tracking process is continued. On the other hand, when the target candidate for tracking is not detected, the target candidate for tracking is deleted from the target candidate.

  The target specifying unit 1g determines whether each target candidate is specified as a target. When a target candidate is continuously tracked by the candidate tracking unit 1f and detected as a target candidate by the target candidate detecting unit 1d for a predetermined period, the target candidate is specified as a target. .

  In such an image target detection and tracking device 1, the image data transmitted from the camera 2 is input by the image input means 1a and stored in the image storage means 1b. The image data stored in the image storage unit 1b is appropriately updated by the image input unit 1a. In addition, a template for detecting a target is stored in advance in the template image storage unit 1c.

  After at least a new input image is acquired at a predetermined timing, the target candidate detection unit 1d and the candidate tracking unit 1f are activated. The target candidate detection unit 1d divides the input image acquired by the image input unit 1a stored in the image storage unit 1b into predetermined units, and calculates the correlation between the divided image and the target template. Then, an object included in a divided image whose calculation result exceeds a predetermined reference value determined in advance is determined as a target candidate. If it is not a registered target candidate, it is newly registered as a target candidate. At this time, the divided image in which the target candidate object is detected is stored in the candidate template image storage unit 1e as a candidate template. The candidate tracking unit 1f divides the input image into predetermined units for each registered target candidate, matches the divided image with the candidate template of the target candidate to be tracked, and the target candidate image to be tracked is obtained. A position on the detected input image is searched. If no tracking target candidate is detected, the registration of the tracking target candidate is deleted. If detected, the position of the target candidate to be tracked is stored and tracking is continued. For each target candidate, the target specifying unit 1g determines whether the target candidate is stably tracked by the candidate tracking unit 1f and is detected as a target candidate by the target candidate detecting unit 1d. If this state continues for a predetermined period, the target candidate is specified as the target. Even after the target is specified, it can be continuously tracked by the candidate tracking unit 1f. However, a tracking unit for the target may be separately provided for tracking.

  In this way, target candidates that satisfy a predetermined criterion are selected from the input image, and tracking of each target candidate is started. Then, the object that could not be tracked is excluded from the target candidates. Objects that appear temporarily are excluded from target candidates because they cannot be tracked continuously. On the other hand, if the target is a detection target, it can be continuously tracked. In this way, an object that can be stably tracked after the start of tracking and is detected as a target candidate by the target template is specified as a target. Thereby, an object having a shape similar to the detection target that appears temporarily in the image can be excluded from the target candidates, and a specific target can be reliably detected and tracked.

  Hereinafter, embodiments will be described in detail by taking as an example an image target detection and tracking device that detects a target such as a lifeboat from a marine image photographed by an infrared camera from a helicopter or the like and tracks the target. In such an image, it is known that an object (such as a wave front) having a shape similar to the target object to be detected appears temporarily.

  First, an image of the sea taken from a distance by an aircraft or the like will be described. FIG. 2 is a diagram illustrating an example of a captured image. (A) is an example of the image of the sea image | photographed from far away, (B) is the figure which showed the location with the high brightness of (A), and is a figure for demonstrating the image of (A).

  As shown in the image of (A), when the sea is photographed from a distance by an aircraft or the like, the detection target such as a lifeboat (the part surrounded by a white circle in the figure) and the wave front around the detection target have high brightness. Is obtained.

  (B) is the figure which expressed the part which has predetermined | prescribed brightness with the circle | round | yen or an ellipse in the image of (A). A detection target 201 indicated by a circle represents a detection target surrounded by a white circle in the image of (A). A wavefront 202 indicated by an ellipse represents a peripheral wavefront in the image of (A). When a target template for detecting the detection target 201 is prepared using this image and target detection is performed by image correlation, the peripheral wave front 202 is detected in addition to the detection target 201.

The image target detection / tracking apparatus 1 according to the embodiment identifies a detection target 201 from a plurality of detection targets 201 and wave fronts 202 detected as target candidates, and performs tracking thereof.
Here, the hardware configuration of the image target detection and tracking apparatus 1 will be described. FIG. 3 is a block diagram illustrating a hardware configuration example of the image target detection and tracking apparatus.

  The entire image target detection / tracking apparatus 1 is controlled by a CPU (Central Processing Unit) 101. A random access memory (RAM) 102, a hard disk drive (HDD) 103, a graphic processing device 104, an input interface 105, and a communication interface 106 are connected to the CPU 101 via a bus 107.

  The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the CPU 101. The RAM 102 stores various data necessary for processing by the CPU 101. The HDD 103 stores the OS and application programs. A monitor 108 is connected to the graphic processing device 104, and an image is displayed on the screen of the monitor 108 in accordance with a command from the CPU 101. A keyboard 109 a and a mouse 109 b are connected to the input interface 105, and signals transmitted from the keyboard 109 a and the mouse 109 b are transmitted to the CPU 101 via the bus 107. The communication interface 106 is connected to a network (not shown) and receives image data from the camera 2 via the network.

With such a hardware configuration, the processing function of the image target detection and tracking apparatus 1 can be realized.
Next, a software configuration for realizing the target candidate detecting unit 1d and the candidate tracking unit 1f of the image target detection and tracking device 1 will be described. FIG. 4 is a diagram illustrating a software configuration example of a main part of the image target detection / tracking apparatus.

  The main part of the image target detection / tracking apparatus 1 includes a storage unit 110, a target candidate detection unit 120, a candidate tracking unit 130, and a target specifying unit 140. Note that the target candidate detection unit 120 and the candidate tracking unit 130 perform matching processing between the divided image obtained by dividing the input image into regions of a predetermined unit and the template. Hereinafter, this divided image is referred to as a block.

  The storage unit 110 is a storage unit that stores various types of information, and stores an input image 111, a target template 112, a candidate template 113, candidate management information 114, and candidate storage memory management information 115. The input image 111 is image data captured by a camera. The target template 112 is image data of a template including a target shape to be detected. The candidate template 113 is image data of a candidate template including a target candidate that is stored in association with the target candidate every time a target candidate is newly registered. In the candidate management information 114, information for managing registered target candidates is set. Information for managing candidate templates used for tracking target candidates to be tracked is set in the candidate storage memory management information 115. Details of each information will be described later.

  The target candidate detection unit 120 is a target candidate detection unit having a template matching processing unit 121 and a candidate information setting unit 122. The template matching processing unit 121 reads the input image 111 and the target template 112 and performs template matching. For example, the difference absolute value between the pixel of the block of the input image and the pixel of the template is calculated, and the sum of the difference absolute value in the block is calculated to calculate the image correlation. Based on the calculation result, an object included in a divided image exceeding a predetermined reference is set as a target candidate. The block in which the target candidate is detected is notified to the candidate information setting unit 122. The candidate information setting unit 122 determines whether the notified target candidate is a newly detected target candidate, and sets the candidate management information 114 according to the determination result. If it is a newly detected target candidate, a new ID is assigned to this target candidate and newly registered in the candidate management information 114. If it is not a newly detected target candidate, a matching registered target candidate is searched, and registration information relating to the searched target candidate is updated. Here, a detection counter indicating that the target candidate has been detected is incremented.

  The candidate tracking unit 130 is a candidate tracking unit that includes a tracking processing unit 131 and a management information update unit 132. The tracking processing of the tracking processing unit 131 is performed by reading the candidate template 113 set for each target candidate, performing pattern matching with the block of the input image 111, and searching for a block in which the target candidate is detected. The tracking result is notified to the management information update unit 132. Based on the tracking result of the tracking processing unit 131, the management information updating unit 132 updates the corresponding information in the candidate management information 114 regarding the target candidate that can be tracked. Here, a tracking counter indicating that the tracking has been performed is incremented. When the tracking cannot be performed, the target candidate is deleted from the candidate management information 114.

  The target specifying unit 140 is a target specifying unit. The candidate management information 114 stored in the storage unit 110 is read, and a target is specified from the registered target candidates based on the candidate management information 114. Here, when the target candidate detection counter and the tracking counter exceed a predetermined threshold, the target candidate is specified as a target.

  Next, management information including candidate management information 114 and candidate storage memory management information 115 will be described. FIG. 5 is a diagram showing an example of management information. (A) is a figure showing an example of candidate management information. (B) is a figure showing an example of candidate storage memory management information.

  The candidate management information of (A) includes candidate number (ID) 1140a, position (coordinates) 1140b in the input image, block number 1140c in the candidate storage memory, tracking count 1140d, detection count 1140e, and the next candidate data. Each pointer 1140f.

  The candidate number (ID) 1140a stores a unique identification number assigned when the target candidate detection unit 120 newly detects a target candidate. The position (coordinates) 1140b in the input image stores coordinates indicating the position on the input image of the image block in which the target candidate is detected. For example, the coordinates of the upper left vertex of each block are set as the position coordinates of the block. The block number 1140c in the candidate storage memory stores the number of the storage memory in which the candidate template used for tracking the target candidate is stored. The tracking count 1140d stores a count value of a counter that is incremented when the candidate tracking unit 130 can track a target candidate to be tracked. The detection count 1140e stores a count value of a counter that is incremented when the target candidate detection unit 120 detects the target candidate. The pointer to the next candidate data 1140f stores a pointer that points to a memory in which candidate management information related to the target candidate of the next ID registered as the target candidate is stored. When this target candidate is the last, a null code (NULL) is set.

  The candidate storage memory management information of (B) includes information items of an index 1150a, coordinate data 1150b of a block storing a candidate template, and a busy flag 1150c.

  The index 1150a indicates a block prepared for storing the image data of the candidate template. Here, for the sake of simplicity, assume that candidate template storage blocks of the same size are prepared corresponding to the block numbers B1, B2,..., Bn (n is an arbitrary integer) of the input image. In order to distinguish from the blocks of the input image, they are denoted as b1, b2,. The block coordinate data 1150b stores coordinates indicating the position of each block (b1, b2,..., Bn). Similar to the block of the input image, the coordinates of the upper left vertex of each block are set as the position coordinates. The in-use flag 1150c is a flag indicating whether or not the corresponding block is used as a candidate template storage area. Set to 1 if in use, 0 if not in use. If no block exists, an empty code is set.

The target candidate detection process, candidate tracking process, and target specifying process are executed with reference to the above management information.
The image target detection tracking process by each processing unit will be described using a specific example. FIG. 6 is a diagram illustrating an example of an input image and a target template.

  In the illustrated example, the input image 1110 is divided into 8 blocks in the row direction and 7 blocks in the column direction. For convenience, the upper left block of the input image 1110 is referred to as block 1 (hereinafter referred to as B1) 1111 and numbers B2, B3,..., B8 are assigned in the row direction. Are sequentially assigned as the numbers next to the last number in all rows.

  In the target candidate detection unit 120, the template matching processing unit 121 calculates the correlation between the image of the target template 1120 having the same size as the block and the image of each block. For example, the correlation with the image of the target template 1120 is calculated from B1 to B56 in the order of block numbers. Then, an object included in a block whose calculation result exceeds a predetermined reference value is set as a target candidate. In the illustrated example, the template matching processing unit 121 detects target candidate 1 from block T1 (1112), target candidate 2 from block T2 (1113), and target candidate 3 from block T3 (1114), and T1 (1112). The block coordinates (x1, y1) of T2, the block coordinates (x2, y2) of T2 (1113), and the block coordinates (x3, y3) of T3 (1114) are notified to the candidate information setting unit 122.

  The candidate information setting unit 122 sets candidate management information 114 and candidate storage memory management information 115 based on the notification. First, description will be continued assuming that all of these target candidates are newly detected target candidates.

  The candidate management information 114 related to the target candidate 1 detected at T1 (1112) will be described. First, ID “1” is assigned to the target candidate 1 and registered in the candidate number (ID) 1140a. The position coordinate “(x1, y1)” of T1 (1112) is registered in the position (coordinates) 1140b in the input image. The image data of T1 (1112) is registered in the candidate storage memory as a candidate template. Here, it is assumed that registration is performed in the block b1 of the candidate storage memory, and “1” is registered in the block number 1140c in the candidate storage memory. Based on the candidate storage memory management information 115, the coordinate data of the block b1 is read out, the image data of T1 (1112) is stored in the corresponding memory area, and the in-use flag 1150c is set to “1” in use. Further, the tracking count 1140d is initialized to “0” and the detection count 1140e is initialized to “1”. When there is next candidate data, the pointer is set. Here, a pointer indicating the candidate management information of the target candidate 2 detected at T2 (1113) is registered in the pointer 1140f to the next candidate data.

Similar processing is performed for the target candidate 2 detected at T2 (1113) and the target candidate detected at T3 (1114).
FIG. 7 is a diagram illustrating an example of management information after the processing of the candidate information setting unit. (A) shows an example of candidate management information, and (B) shows an example of candidate storage memory management information.

  The example of candidate management information shown in (A) has a structure in which candidate management information 1141 for target candidate 1, candidate management information 1142 for target candidate 2, and candidate management information 1143 for target candidate 3 are linked by pointers. ing.

  The candidate management information 1141 is as described above. In the figure, the tracking count and the detection count indicate a state updated from the initial state described above. The candidate management information 1142 includes a target candidate ID “2”, a position coordinate “(x2, y2)” of T2 (1113), a block number “3” in the candidate storage memory, a tracking count and a detection count, A pointer for indicating candidate candidate candidate management information is stored. The candidate management information 1143 stores the target candidate ID “3”, the position coordinate “(x3, y3)” of T3 (1114), the block number “4” in the candidate storage memory, and the tracking count and detection count. Has been. Since there is no next target candidate, an empty code is set for the pointer to the next candidate data.

  In the candidate storage memory management information 1150 shown in (B), the candidate storage memory of the block number registered in the block number in the candidate storage memory of the candidate management information 1141, 1142, 1143 in (A) is in use. It has been shown. Specifically, the block b1 storing the candidate template image data of the target candidate 1, the block b3 storing the candidate template image data of the target candidate 2, and the candidate template image data of the target candidate 3 are stored. The in-use flag corresponding to the block b4 is set to in use (“1”).

  Next, processing performed by the candidate tracking unit 130 from the state of the candidate management information 1141, 1142, 1143 will be described. The candidate tracking unit 130 sequentially performs target candidate tracking processing by following the link information in order from the candidate management information 1141 of the target candidate 1. Then, the process ends when an empty code is detected in the pointer to the next candidate data.

  The case of target candidate 1 will be described. The tracking processing unit 131 reads the candidate management information 1141 of the target candidate 1 and determines the search range of the target candidate 1 based on the position (coordinates) “(x1, y1)” in the input image. The search range is determined empirically in accordance with the period of updating the input image, the period of tracking processing, the estimated moving speed of the object to be detected, and the like. Further, the moving speed of the object to be detected may be measured from the image, and the detection range may be appropriately adjusted based on the measurement result. For example, when a block adjacent to a block in which a target candidate is detected is defined as a range, T1 (1112) in which target candidate 1 is detected and its neighboring blocks B9, B10, B11, B17, B19, B25, B26, B27 becomes the search range. The candidate image of the block in the search range is matched with the candidate template of the target candidate 1 indicated by the block number in the candidate storage memory of the candidate management information 1141, and the target candidate 1 is detected. The candidate template in this case is T1 (1112) image data stored in the block b1 of the candidate storage memory when the target candidate 1 is detected. When the target candidate 1 is detected, the management information update unit 132 is notified of the detected block number or its coordinates, and when not detected, the management information update unit 132 is notified of the detection failure. When the target candidate 1 is detected, the management information update unit 132 registers the notified block coordinates in the position (coordinates) in the input image of the candidate management information 1141 and increments the tracking count. When the target candidate 1 is not detected, the registration of the target candidate 1 is deleted. For example, the pointer indicating the candidate management information 1141 of the target candidate 1 is changed to the candidate management information 1142 of the target candidate 2. In this example, the target candidate is immediately deleted because it cannot be detected. However, the target candidate may be deleted when it cannot be detected for a predetermined period. Similar processing is performed for the target candidate 2 and the target candidate 3, and the candidate management information 1142 and 1143 are updated.

  Next, the process of the target candidate detection unit 120 will be described. Using the target template 1120, the template matching processing unit 121 sets an object included in a block whose correlation with the target template 1120 exceeds a predetermined reference value as a target candidate, and the coordinates of the detected block are the candidate information setting unit 122. It is the same as the above to be notified. When the candidate management information 114 is registered, the candidate information setting unit 122 collates the notified block coordinates with the position (coordinates) in the input image related to the already registered target candidate. When matching candidate management information is detected, it is determined that this target candidate has been detected, and the detection count of this candidate management information is incremented. On the other hand, when no matching candidate management information is detected, a new target candidate is set and registered in the candidate management information.

  The above processing is performed and the candidate management information is updated. The target identifying unit 140 reads the candidate management information and determines whether the detection count and the tracking count exceed a predetermined threshold for each target candidate. Then, target candidates that exceed a predetermined threshold are identified as targets. Note that a criterion for specifying a target candidate as a target can be appropriately set according to a detection target. For example, a determination rule such as when the ratio between the detection count and the tracking count exceeds a certain value, or a determination rule such as making the determination criterion variable according to conditions may be set.

FIG. 8 is a diagram showing a flow of processing up to target specification together with image changes of target candidates.
In the figure, time 1, time 2, time 3, and time 4 have elapsed. The image shows a change in the image of each target candidate along with the elapsed time. For example, the image of target candidate 1 is in the state of image 301 at time 1, image 311 at time 2, image 321 at time 3, and image 331 at time 4. The image of the target candidate 2 is in the state of an image 302 at time 1, an image 312 at time 2, an image 322 at time 3, and an image 332 at time 4. The target candidate 3 is in the state of an image 303 at time 1, an image 313 at time 2, and an image 323 at time 3.

  First, it is assumed that target candidate detection unit 120 detects target candidate 1, target candidate 2, and target candidate 3 for the first time at time 1. The search for the target candidate at this time is performed using the target template 1120. At this time, the images in which the target candidates are detected, that is, the target candidate 1 image 301, the target candidate 2 image 302, and the target candidate 3 image 303 are registered as candidate templates for the respective target candidates.

  The operation at time 2 will be described. The image 301 of the target candidate 1 at time 1 changes to the state of the image 311 at time 2, but is detected as the target candidate 1 by the candidate tracking unit 130. The image 302 of the target candidate 2 at time 1 is little changed from the state of the image 312 at time 2 and is detected as the target candidate 2 by the candidate tracking unit 130. Then, the image 303 of the target candidate 3 at time 1 changes to the state of the image 313 at time 2, but is detected as the target candidate 3 by the candidate tracking unit 130.

  The operation at time 3 will be described. The image 311 of the target candidate 1 at time 2 changes to the state of the image 321 at time 3, but is detected as the target candidate 1 by the candidate tracking unit 130. The image 312 of the target candidate 2 at time 2 has little change in the state of the image 322 at time 3 and is detected as the target candidate 2 by the candidate tracking unit 130. Then, since the image 313 of the target candidate 3 at time 2 changes to the state of the image 323 at time 3, the candidate tracking unit 130 cannot detect the target candidate 3. As a result, tracking becomes impossible and the target candidate 3 is deleted from the target candidates.

  The operation at time 4 will be described. The image 321 of the target candidate 1 at time 3 changes to the state of the image 331 at time 4, and the candidate tracking unit 130 cannot detect the target candidate 1. For this reason, tracking becomes impossible and target candidate 1 is also deleted from the target candidates. The image 322 of the target candidate 2 at time 3 has little change in the state of the image 332 at time 4 and is detected as the target candidate 2 by the candidate tracking unit 130.

  By performing the above processing, the target candidates (target candidate 1 and target candidate 3) that are temporarily generated due to a wavefront or the like disappear as the wavefront disappears and are also removed from the target candidates. Then, the target candidate 2 detected continuously and stably is specified as the target.

  In the above description, the operation of the target candidate detection unit 120 is omitted. However, by adding the detection result of the target candidate detection unit 120, the target can be specified more accurately.

Next, an image target detection and tracking method performed by the image target detection and tracking apparatus will be described using a flowchart.
FIG. 9 is a flowchart showing the entire processing procedure of the image target detection and tracking method.

Processing is started at a predetermined cycle.
[Step S01] Image data captured by the camera is input.
[Step S02] The candidate management information is read out, and it is checked whether or not the target candidate is registered in the candidate management information. If the target candidate is registered, the process proceeds to step S03. If the target candidate is not registered, the process proceeds to step S04.

[Step S03] When the target candidate is registered, the target candidate is detected from the input image based on the candidate management information, and tracking processing for tracking the target candidate is performed. Details will be described later.
[Step S04] When the target candidate is not registered, or when the target candidate tracking process is completed, the target candidate detection process for detecting the target candidate from the input image is performed. Details will be described later.

[Step S05] Based on the target candidate tracking process in step S03 and the target candidate detection process in step S04, a target is specified from the target candidates.
Each process will be described.

FIG. 10 is a flowchart showing a processing procedure for target candidate tracking processing.
[Step S31] A target candidate n (n is an arbitrary integer) to be processed is initialized to a target candidate 1.

  [Step S32] The target candidate n is tracked. Specifically, based on the candidate management information, an image of a peripheral block indicated by a position (coordinates) in the input image where the target candidate n is detected, and a candidate template indicated by a block number in the candidate storage memory The target candidate is tracked by matching with the image.

[Step S33] It is determined whether the target candidate is traceable. If tracking is possible, the process proceeds to step S34. If tracking is impossible, the process proceeds to step S35.
[Step S34] If the target candidate is traceable, the tracking count of the target candidate n is incremented, and the process proceeds to step S36.

[Step S35] When the target candidate cannot be tracked, the target candidate n is deleted from the target candidates.
[Step S36] It is determined whether all candidates have been completed. When finished, the process ends. If not completed, n is incremented by 1, and the process returns to step S32 to process the next target candidate.

  The target candidate is tracked by executing the above processing procedure. Then, the target candidate that cannot be tracked is deleted from the candidates. Thereby, an object having a shape similar to a detection target that occurs temporarily can be excluded from the target.

FIG. 11 is a flowchart showing a processing procedure of target candidate detection processing.
[Step S41] The block number m (m is an arbitrary integer) of the input image to be processed is initialized to B1.

[Step S42] The correlation between the image of the block Bm of the input image and the image of the target template is calculated.
[Step S43] It is determined whether the correlation value calculated in step S42 is greater than or equal to a reference value. If it exceeds the reference value, the process proceeds to step S44. If it does not exceed the reference value, the process proceeds to step S48.

  [Step S44] When the correlation value exceeds the reference value, this block number m is collated with the position (coordinates) in the input image of the already-targeted candidate already registered in the candidate management information. If they match, it is determined that the images are matched target candidates. If they do not match, it is determined to be a new candidate.

  [Step S45] It is determined whether the object of the block Bm is a new candidate. If it is a new candidate, the process proceeds to step S46. If it is not a new candidate, the process proceeds to step S47.

[Step S46] If the object in the block Bm is a new candidate, registration is added to the candidate management information as a candidate candidate, and the process proceeds to step S48.
[Step S47] When the object of the block Bm is not a new candidate, the detection count of the target candidate having the same position (coordinates) in the input image as the block m detected in step S43 is incremented.

  [Step S48] It is determined whether all blocks of the input image have been completed. When finished, the process ends. If not completed, m is advanced by 1, and the process returns to step S42 to process the next block.

  By executing the above processing procedure, a new target candidate is detected. For registered target candidates, whether or not matching with the target template can be detected is detected.

FIG. 12 is a flowchart showing the processing procedure of the target specifying process.
[Step S51] The target candidate n to be processed is initialized to the target candidate 1.
[Step S52] The detection count and tracking count of the target candidate n are read from the candidate management information related to the target candidate n.

  [Step S53] The detected detection count and the tracking count are evaluated. For example, it is evaluated whether the detection count and the tracking count exceed a predetermined threshold.

  [Step S54] As a result of the evaluation in step S53, it is determined whether the target candidate n can be specified as a target. If the target can be specified, the process proceeds to step S55. If the target cannot be specified, the process proceeds to step S56.

[Step S55] When the target candidate n can be specified as the target, the target candidate n is set as the target.
[Step S56] It is determined whether all candidates have been completed. When finished, the process ends. If not completed, n is advanced by 1, and the process returns to step S52 to process the next target candidate.

By performing the above processing procedure, target candidates that can be continuously tracked and detected stably are identified as targets.
The above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the image target detection and tracking apparatus should have is provided. By executing the program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic recording device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical disks include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc Read Only Memory), CD-R (Recordable) / RW (ReWritable), and the like. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, portable recording media such as a DVD and a CD-ROM in which the program is recorded are sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

(Supplementary Note 1) In an image target detection and tracking device that detects and tracks a target from an input image,
The template is read from a template storage unit that stores a template image including a target object, a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit is calculated, and the calculation result is a predetermined reference value Target candidate detection means for registering an object included in the divided image exceeding the target candidate;
For each target candidate, with the target candidate as a tracking target, the tracking included in the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate The target candidate for the target is searched, and when the target candidate for the tracking target can be searched, the tracking of the target candidate for the tracking target is continued, and when the target candidate for the tracking target cannot be searched, the target of the tracking target Candidate tracking means for deleting from the target candidates,
A target that identifies the target candidate as a target when the target candidate is continuously tracked by the candidate tracking unit for a predetermined period and is detected as the target candidate by the target candidate detection unit Specific means,
An image target detection and tracking device comprising:

(Additional remark 2) The said target candidate detection means stores the said divided image in which the said object registered into the said target candidate was contained in a candidate template memory | storage means as a candidate template,
The candidate tracking unit reads the candidate template stored in the candidate template storage unit, and based on the correlation between the divided image obtained by dividing the most recent input image by a predetermined unit and the candidate template, the target to be tracked Search for candidates,
The image target detection and tracking device according to appendix 1, wherein:

  (Additional remark 3) The said candidate tracking means is a range previously determined centering on the position on the said input image of the said division | segmentation image from which the target candidate of the said tracking target was detected the search range of the target candidate of the said tracking target The image target detection and tracking device according to appendix 2, wherein the target candidate is searched only for the divided image.

(Supplementary Note 4) The candidate tracking unit stores a position on the input image of the divided image where the target candidate of the tracking target is detected in association with the target candidate of the tracking target,
When the target candidate detecting means detects the target candidate, the position on the input image of the divided image from which the target candidate is detected is detected by the candidate tracking means. It is determined whether or not the position of the divided image matches the position on the input image. When there is a target candidate for tracking that matches the position, the target candidate is determined to be the target candidate for tracking. When there is no tracking target candidate whose position matches, the target candidate is registered as a new target candidate.
The image target detection and tracking device according to Supplementary Note 2, wherein

(Additional remark 5) The said target candidate detection means updates the detection counter regarding this target candidate, whenever the said target candidate is detected,
The candidate tracking unit updates a tracking counter related to the target candidate every time the target candidate to be tracked is detected,
The target specifying unit reads out the value of the detection counter and the value of the tracking counter for the target candidate for each target candidate, and the value of the detection counter and the value of the tracking counter exceed a predetermined threshold value. When the target candidate is identified as the target,
The image target detection and tracking device according to appendix 1, wherein:

(Additional remark 6) In the image target detection tracking apparatus which detects and tracks a target from an input image,
A target candidate detecting unit reads the template from a template storage unit that stores a template image including a target object, calculates a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit, and calculates Register an object included in the divided image whose result exceeds a predetermined reference value as a target candidate,
For each target candidate, the candidate tracking means uses the target candidate as a tracking target, the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate When the target candidate for tracking included in the search target candidate is searched and the target candidate for tracking target can be searched, the tracking of the target candidate for tracking target is continued, and when the target candidate for tracking target cannot be searched Deleting the target candidate to be tracked from the target candidate;
When the target identification means is continuously tracked by the candidate tracking means for a predetermined period determined in advance and is detected as the target candidate by the target candidate detection means, the target candidate is Identify as goals,
An image target detection and tracking method comprising a procedure.

(Supplementary Note 7) In an image target detection and tracking program for detecting and tracking a target from an input image,
Computer
The template is read from a template storage unit that stores a template image including a target object, a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit is calculated, and the calculation result is a predetermined reference value Target candidate detecting means for registering an object included in the divided image exceeding the target candidate,
For each target candidate, with the target candidate as a tracking target, the tracking included in the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate The target candidate for the target is searched, and when the target candidate for the tracking target can be searched, the tracking of the target candidate for the tracking target is continued, and when the target candidate for the tracking target cannot be searched, the target of the tracking target Candidate tracking means for deleting candidates from the target candidates;
A target that identifies the target candidate as a target when the target candidate is continuously tracked by the candidate tracking unit for a predetermined period and is detected as the target candidate by the target candidate detection unit Specific means,
An image target detection and tracking program characterized in that it functions as a computer program.

It is a block diagram of the image target detection tracking apparatus of embodiment. It is the figure which showed an example of the picked-up image. It is the block diagram which showed the hardware structural example of the image target detection tracking apparatus. It is the figure which showed the software structural example of the principal part of an image target detection tracking device. It is the figure which showed an example of management information. It is the figure which showed an example of the input image and the target template. It is the figure which showed an example of the management information after the process completion of a candidate information setting part. It is the figure which showed the flow of the process until target specification with the image change of the target candidate. It is the flowchart which showed the whole processing procedure of the image target detection tracking method. It is the flowchart which showed the process sequence of the target candidate tracking process. It is the flowchart which showed the process sequence of the target candidate detection process. It is the flowchart which showed the process sequence of the target specific process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image target detection tracking device 1a Image input means 1b Image storage means 1c Template image storage means 1d Target candidate detection means 1e Candidate template image storage means 1f Candidate tracking means 1g Target specification means 2 Camera

Claims (5)

In an image target detection and tracking device that detects and tracks a target from an input image,
The template is read from a template storage unit that stores a template image including a target object, a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit is calculated, and the calculation result is a predetermined reference value Target candidate detection means for registering an object included in the divided image exceeding the target candidate;
For each target candidate, with the target candidate as a tracking target, the tracking included in the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate The target candidate for the target is searched, and when the target candidate for the tracking target can be searched, the tracking of the target candidate for the tracking target is continued, and when the target candidate for the tracking target cannot be searched, the target of the tracking target Candidate tracking means for deleting candidates from the target candidates;
A target that identifies the target candidate as a target when the target candidate is continuously tracked by the candidate tracking unit for a predetermined period and is detected as the target candidate by the target candidate detection unit Specific means,
An image target detection and tracking device comprising: The target candidate detection unit stores the divided image that includes the object registered in the target candidate as a candidate template in a candidate template storage unit,
The candidate tracking unit reads the candidate template stored in the candidate template storage unit, and based on the correlation between the divided image obtained by dividing the most recent input image by a predetermined unit and the candidate template, the target to be tracked Search for candidates,
The image target detection and tracking device according to claim 1. The candidate tracking means stores a position on the input image of the divided image where the target candidate for the tracking target is detected in association with the target candidate for the tracking target,
When the target candidate detecting means detects the target candidate, the position on the input image of the divided image from which the target candidate is detected is detected by the candidate tracking means. It is determined whether or not the position of the divided image matches the position on the input image. When there is a target candidate for tracking that matches the position, the target candidate is determined to be the target candidate for tracking. When there is no tracking target candidate whose position matches, the target candidate is registered as a new target candidate.
The image target detection and tracking device according to claim 2, wherein: In an image target detection and tracking device that detects and tracks a target from an input image,
A target candidate detecting unit reads the template from a template storage unit that stores a template image including a target object, calculates a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit, and calculates Register an object included in the divided image whose result exceeds a predetermined reference value as a target candidate,
For each target candidate, the candidate tracking means uses the target candidate as a tracking target, the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate When the target candidate for tracking included in the search target candidate is searched and the target candidate for tracking target can be searched, the tracking of the target candidate for tracking target is continued, and when the target candidate for tracking target cannot be searched Deleting the target candidate to be tracked from the target candidate;
When the target identification means is continuously tracked by the candidate tracking means for a predetermined period determined in advance and is detected as the target candidate by the target candidate detection means, the target candidate is Identify as goals,
An image target detection and tracking method comprising a procedure. In an image target detection and tracking program for detecting and tracking a target from an input image,
Computer
The template is read from a template storage unit that stores a template image including a target object, a correlation between the template and a divided image obtained by dividing the input image by a predetermined unit is calculated, and the calculation result is a predetermined reference value Target candidate detecting means for registering an object included in the divided image exceeding the target candidate,
For each target candidate, with the target candidate as a tracking target, the tracking included in the input image in which the target candidate is newly detected or the latest input image taken after the input image used for tracking the target candidate The target candidate for the target is searched, and when the target candidate for the tracking target can be searched, the tracking of the target candidate for the tracking target is continued, and when the target candidate for the tracking target cannot be searched, the target of the tracking target Candidate tracking means for deleting candidates from the target candidates;
A target that identifies the target candidate as a target when the target candidate is continuously tracked by the candidate tracking unit for a predetermined period and is detected as the target candidate by the target candidate detection unit Specific means,
An image target detection and tracking program characterized in that it functions as a computer program.

Images