JP4768544B2 - Target detection tracking device - Google Patents

Description

  The present invention performs image processing from a video signal supplied from an imaging device such as a television camera or an infrared imaging device, determines a tracking target from the feature amount or positional relationship of an object image in two consecutive images, and the imaging device This invention relates to a target detection and tracking device that tracks a tracking target by controlling the orientation of the target.

  High accuracy is required for guiding bullets. Conventionally, a target detection tracking device that tracks a tracking target using image data captured by an imaging device has been used for guiding a guided bullet.

  Specifically, this target detection tracking device converts an image signal supplied from an imaging device such as a television camera or an infrared imaging device into a digital signal, detects an object image in the image, and labels each object image. Assign, create a label table, extract features such as the maximum value, minimum value, coordinates, area, etc. from the object image for each label, and control the direction of the imaging device based on these information and track Follow the goal.

  In the conventional target detection and tracking device, the label table is created independently for each captured frame, so even in the same tracking target image, the same label is not always attached to different frames, and time Cannot be used as data indicating the movement of a continuous tracking target.

Patent Document 1 describes that this problem can be solved by labeling an object image once for each frame and then performing labeling so that the same target label is attached to a tracking target candidate image between multiple frames. Has been.
Japanese Patent No. 3666892

  However, in the technique described in Patent Document 1, it is difficult to determine the tracking target in the current frame when a plurality of tracking target candidates in the current frame overlap the tracking target of the previous frame or when there is no overlap at all. There was a problem.

  The present invention has been made in view of the above-described conventional problems, and tracking is performed even when a plurality of tracking target candidates of the current frame overlap the tracking target of the previous frame or when there is no overlap at all. It is an object of the present invention to provide an object detection and tracking device capable of determining a target.

In order to achieve the above object, the invention of claim 1 includes an AD conversion unit that converts a video signal obtained by the imaging processing unit into an image signal of a digital signal for each frame, and an image in the image converted by the AD conversion unit. A labeling processing unit that assigns a label to each object image, an area calculation storage unit that calculates an area of the object image to which the labeling processing unit assigns a label, and stores a feature amount table that stores the area for each label, and a previous frame A link investigation processing unit that detects a positional relationship between the object image in the current frame and the object image in the current frame, and an area for each label of the object image in the previous frame and the object image in the current frame stored in the feature amount table by the area calculation storage unit. from correlation and coupling survey processing unit positional relationship by label detected, tracked target time frame that was present in the previous frame The generated object image has an area highly correlated with the area of the tracking target in the previous frame among the generated object images when it is determined that the object image has been generated within a certain distance from the position of the tracking target in the previous frame. and the object image and determines a target determination unit of the tracking target in the frame time, based on the position of the tracking target target determining unit determines, target detection tracking device comprising a gimbal mechanism to control the direction of the imaging processing unit, the I will provide a.

  According to the present invention, it is possible to determine a tracking target even when a plurality of tracking target candidates in the current frame overlap the tracking target in the previous frame or when there is no overlap at all.

  Hereinafter, embodiments of a target detection tracking device according to the present invention will be described with reference to the drawings.

<Embodiment 1>
FIG. 1 shows the configuration of the first embodiment of the present invention. The target detection and tracking device captures an imaging field of view at a predetermined frame rate, sequentially outputs the imaging frames, and an AD conversion unit 102 that converts an output signal of the imaging processing unit 101 into a digital signal. The AD converter 102 receives a digitized video signal and performs necessary preprocessing, a feature enhancement unit 104 that enhances the contour of the image, and binarizes the signal with enhanced contour. The binarization processing unit 105 that extracts the emphasized portion of the image, the labeling processing unit 107 that assigns a label to each object image that is the portion extracted by the binarization processing unit 105, and the feature enhancement unit 104 And a feature amount calculation storage unit 106 for calculating and storing a feature amount for each object image to which a label is attached from the output of the labeling processing unit 107, and before the label is attached. A connection investigation processing unit 108 that investigates a positional relationship such as an overlap of the current object image, a feature amount of each object image calculated by the feature amount calculation storage unit 106, and a position of each object detected by the connection investigation processing unit 108 A target determination unit 109 that determines a tracking target from the relationship; a tracking processing unit 110 that creates information about the target from the output of the target determination unit 109; and a gimbal that controls the direction of the imaging device based on the information output from the tracking processing unit 110 And a mechanism unit 111.

  In the labeling processing unit 107, the binarization processing unit 105 labels the obtained object images in order, and in the concatenation investigation processing unit 108, the object image of the previous frame and the current frame to which the labeling processing unit 107 has been given a label. From the coordinates of the object image, the positional relationship such as the overlap of both object images is investigated.

  Here, the positional relationship includes, for example, overlapping of object images, but is not limited to this and includes a relative positional relationship based on a certain object image.

  On the other hand, the feature amount of the object image feature-enhanced by the feature enhancement unit 104 is calculated and stored in the feature amount calculation storage unit 106. Here, the feature enhancement includes, for example, contour correction, but is not limited thereto. The target determination unit 109 calculates the current frame based on the positional relationship between the object image value in the previous frame calculated by the feature amount calculation storage unit 106 and the feature value of the object image in the current frame and the object image detected by the connection investigation processing unit 108. The object image of the tracking target in is determined.

  Next, the processing in each part of the target detection tracking device will be described more specifically.

  A flowchart of the labeling process in the labeling processing unit 107 is shown in FIG. In step S <b> 201, the labeling processing unit 107 monitors the input of one frame image from the binarization processing unit 105.

  When one frame is input from the binarization processing unit 105 to the labeling processing unit 107, in step S202, the labeling processing unit 107 detects an object image in the frame. In step S203, the labeling processing unit 107 detects the frame. The labels are sequentially attached to the detected object images. In step S204, the labeling processing unit 107 stores the label and position in the object image table.

  In step S205, the labeling processing unit 107 determines whether there is an object image to which no label is added in the frame. If there is an object image to which no label is assigned, the process returns to step 202. If not, the object image table storing the label and position is sent to the feature amount calculation storage unit 106 and the connection investigation processing unit 108 in step S206. In this embodiment, the feature amount calculation storage unit 106 is specifically an area calculation storage unit that calculates the area of an object image.

  Next, processing in the feature quantity calculation storage unit 106 will be described using the flowchart shown in FIG.

  The feature quantity calculation storage unit 106 monitors inputs from the feature enhancement unit 104 and the labeling processing unit 107 in step S301. When there is an input from the feature enhancement unit 104 and the labeling processing unit 107, in step S302, the feature amount calculation storage unit 106 extracts a label from the object image table in which the label and position input from the labeling processing unit 107 are stored. And read position. In the next step S303, the feature amount calculation processing unit 106 calculates the area of the object image existing at the read position. In this embodiment, the feature amount is an area. The feature amount table will be described later.

  Specific examples of the position include, but are not limited to, coordinates, latitude, longitude, and the like.

  In the next step S304, it is determined whether or not there is an object image whose feature value has not been calculated. If there is an object image for which the feature amount has not been calculated, that is, if there is an object image that has been labeled in the object image table but has not been calculated for the feature amount, that is, if yes in step S304, step S302 is performed. Returning to FIG. 5, the label and position of an object image whose feature values have not yet been calculated in the object image table are read, each feature value of this object image is calculated, and those feature values are stored in the feature value table.

  On the other hand, for all labels stored in the object image table in which the label and position input from the labeling processing unit 107 are stored, calculation of the area as the feature amount and storage in the feature amount table are completed. That is, in the case of No in step S304, the completed feature amount table is sent to the target determination unit 109 in step S305.

  The connection investigation processing unit 108 receives the output of the labeling processing unit 107 and performs processing according to the flowchart shown in FIG.

  The connection investigation processing unit 108 constantly monitors the input from the labeling processing unit 107 in step S401. When the object image table is input from the labeling processing unit 107, the label and position are sequentially read from the object image table in step S402.

  Next, in step S403, the connection investigation processing unit 108 determines whether the object image of the current frame overlaps the tracking target of the previous frame, based on the label position corresponding to the read object image of the current frame. If the tracking targets do not overlap, the process returns to step S402 to read the label and position for the next object image. If the object images overlap in the current frame and the previous frame, in step S404, the previous frame, the label and position of the current frame are stored in a memory (not shown).

  In the next step S405, it is checked whether there is an unjudged label, that is, an object image. If there is a label that has not been determined, the process returns to step S402, the label and position are read, and it is checked in step S403 whether the object images overlap.

  On the other hand, if there are no object images for which the overlap state between the current frame and the previous frame has not been determined, the process proceeds to the next step S406, and the overlap state information is sent to the target determination unit 109, and then the process returns to step S401. Prepare for the next input from the labeling processing unit 107.

  The target determination unit 109 is based on the feature amount of each object image input from the feature amount calculation storage unit 106 and the information on the overlapping state of the object images input from the connection investigation processing unit 108, and is a flowchart shown in FIG. To determine the continuity of the tracking target. FIG. 7 is a diagram illustrating the relationship between the tracking target of the previous frame and the object image of the current frame.

  The target determination unit 109 constantly monitors the inputs sent from the feature amount calculation storage unit 106 and the connection survey processing unit 108 in step S501. When there is an input from the feature amount calculation storage unit 106 and the connection survey processing unit 108, the target determination processing unit 109 reads the input from the connection survey processing unit 108 in step S501.

  In step S502, the target determination unit 109 determines whether there is a tracking target in the previous frame. If not, the process proceeds to step S503, and if present, the process proceeds to step S504.

  In step S503, the target determination unit 109 determines whether there is an object image having a feature amount highly correlated with the feature amount of the tracking target stored in advance for the current frame input from the connection survey processing unit 108, or three frames or more before. It is determined whether there is an object image having a feature quantity highly correlated with the feature quantity of the tracking target before disappearance for the tracking target that exists in the previous frame and disappeared in the previous frame (case 1 in FIG. 7). If not, the process returns to step S501. If there is, in step S505, the target determination unit 109 sets an object image having a high correlation in the current frame as a tracking target, and proceeds to step S511.

  Here, in the first embodiment, the feature amount is the area of the object. Therefore, a high correlation between feature amounts means that the areas are equal or approximate.

  In step S504, the target determination unit 109 determines whether there is an object image that overlaps the tracking target of the previous frame in the current frame. If there is, the process proceeds to step S506, and if not, the process proceeds to step S509.

  In step S506, the target determination unit 109 determines whether there are two or more object images of the current frame that overlap the tracking target of the previous frame. If there are no two or more overlapping object images, that is, if there is only one overlapping object image, the process proceeds to step S508. In step S508, the target determination unit 109 sets the object image of the overlapping current frame as a tracking target, and proceeds to step S511.

  In this case, for example, there are the following cases. The case where there is only one object image in the previous frame and there is only one object image in the current frame (case 2 in FIG. 7). When there are one or more object images in addition to the tracking target in the previous frame and these object images are all combined into one in the current frame (case 5 in FIG. 7).

  If there are two or more overlapping object images, the process proceeds to step S507. In step S507, the target determination unit 109 sets an object image having a feature amount highly correlated with the feature amount of the tracking target of the previous frame among the overlapping object images of the current frame, and proceeds to step S511. In this case, for example, there are the following cases. When the object image is only one tracking target in the previous frame, and the tracking target is separated into two or more in the current frame (case 3 in FIG. 7). When there are one or more object images in addition to the tracking target in the previous frame, and some of these object images are combined and separated in the current frame (case 4 in FIG. 7).

  In step 509, the target determination unit 109 determines whether the tracking target that existed in the previous frame has disappeared in the current frame and an object image has been generated within a certain distance from the position of the tracking target in the previous frame. If not, the process returns to step S501. If it has occurred, the process proceeds to step S510, and the target determination unit 109 sets an object image having a feature quantity highly correlated with the feature quantity of the tracking target of the previous frame among the generated object images as a tracking target (Case 6 in FIG. 7). The process proceeds to S511.

  In step S511, the target determination unit 109 sends the tracking target label and position of the current frame to the tracking processing unit 110.

  As described above, in the first embodiment, the target determination unit 109 overlaps the area of each object image calculated by the feature amount calculation storage unit 106 with the tracking target of the previous frame determined by the connection investigation processing unit 108. Therefore, since the tracking target of the current frame is determined, tracking with higher accuracy becomes possible.

<Embodiment 2>
The configuration of the second embodiment is the same as the configuration of the first embodiment. In the second embodiment, one or more types of feature amounts are selected from the area of the object image, the maximum luminance, the barycentric coordinates, and the movement direction, but the feature amounts are not limited to these.

  Processing in the feature quantity calculation storage unit 106 will be described with reference to the flowchart shown in FIG.

  The feature quantity calculation storage unit 106 monitors inputs from the feature enhancement unit 104 and the labeling processing unit 107 in step S701. When there is an input from the feature enhancement unit 104 and the labeling processing unit 107, in step S702, the feature amount calculation storage unit 106 extracts a label from the object image table in which the label and position input from the labeling processing unit 107 are stored. And read position. In the next step S703, the feature amount calculation processing unit 106 calculates an area which is one of the feature amounts of the object image existing at the read position.

  In the next step S704, the luminance, which is one of the feature values of the object image, is calculated. In step S705, the center of gravity, which is one of the feature values of the object image, is calculated. In step S706, the feature value of the object image is calculated. One moving direction is calculated. These calculated feature amounts (area, luminance, center of gravity, moving direction) are stored in a feature amount table corresponding to the label attached to the object image in step S707. Note that the order of the feature quantities to be calculated may be any order.

  In the next step S708, it is determined whether or not there is an object image whose feature value has not been calculated. If there is an object image for which the feature amount has not been calculated, that is, if there is an object image that has been labeled in the object image table but has not been calculated for the feature amount, that is, if yes in step S708, step S702 is performed. Returning to FIG. 5, the label and position of an object image whose feature values have not yet been calculated in the object image table are read, each feature value of this object image is calculated, and those feature values are stored in the feature value table.

  On the other hand, for all the labels stored in the object image table in which the label and position input from the labeling processing unit 107 are stored, calculation of the feature quantity including the area and storage in the feature quantity table are completed. In other words, in the case of No in step S708, the completed feature amount table is sent to the target determination unit 109 in step S709.

  FIG. 8 summarizes representative feature amounts and methods for determining a tracking target for each feature amount. In any feature amount, an object image having a high correlation between the feature amount of the tracking target in the previous frame and the feature amount of the object image in the current frame is set as the tracking target.

  The area can be obtained from the number of dots or the number of pixels in the object image. For example, when an object image having an area of 10 in the previous frame and an object image having an area of 11 and an object image having an area of 11 are detected in the current frame, the object image having an area of 11 is set as the tracking target.

  The luminance can be easily obtained because it is converted into digital data such as RGB by the AD converter 102. When there are a plurality of object images in the current frame, tracking is performed on the object image having the maximum luminance close to the tracking target maximum luminance in the previous frame.

  The center of gravity can be obtained by a known method. When there are a plurality of object images in the current frame, an object image having a centroid position close to the centroid position in the previous frame is set as a tracking target. Specific examples of the position of the center of gravity include, but are not limited to, coordinates of the center of gravity, latitude, and longitude.

  For example, the movement can be obtained by comparing the center of gravity position of the object image with the previous frame and the current frame. Here, the movement may be, for example, a moving direction, but is not limited thereto. When there are a plurality of object images in the current frame, an object image showing a motion that is the same as or similar to the motion in the previous frame is set as a tracking target.

The processing flow of each unit is the same as that in the first embodiment, but the feature amount table created by the feature amount calculation processing unit 106 is as shown in FIG. 9 by increasing the feature amount as described above. A feature table is created for each frame.

  FIG. 9A shows an example of the first frame (previous frame). The label assigned by the labeling processing unit 107 is stored in the label column. For each label, an area, luminance, barycentric position, moving direction, and the like, which are feature amounts, are calculated and stored. In the example of the data of each label, the label is F1L1, the area is 10, the luminance is 20, the center of gravity is (50, 150), and the direction is 0. FIG. 8B shows an example of the second frame (current frame). In the example of the data of each label, the area is 11, the luminance is 19, the barycentric position is (52, 149), and the direction is 3 in the case of the object image with the label F2L1. In the case of an object image with the label F2L2, the area is 5, the luminance is 11, the center of gravity is (90, 45), and the direction is 85. The feature amount table may be stored in a memory or may be stored in an external storage device such as a magnetic disk drive.

  Note that a high correlation between feature amounts means that the feature amounts are the same or approximate. For example, assuming that the label F1L1 in FIG. 9A is a tracking target, referring to the above-described example of data, the label F2L1 in FIG. 9B has a higher correlation of feature amounts than the label F2L2. I understand.

  When calculating the level of correlation, the feature amount may be multiplied by a coefficient so that a particular feature amount is particularly emphasized.

  As described above, the second embodiment has an advantage that tracking can be performed with higher accuracy by using a combination of the area of the object image and another feature amount such as luminance.

  In the present invention, it is possible to calculate one type or two or more of the area, maximum luminance, barycentric coordinates, and moving direction of the object image as the feature amount.

It is the figure which represented typically the structure of the target detection tracking apparatus in embodiment of this invention. It is a flowchart of the labeling process performed in the labeling process part. 5 is a flowchart of a feature amount calculation storage process performed in a feature amount calculation storage unit 106. It is a flowchart of the connection investigation process performed in the connection investigation process part. 4 is a flowchart of a target determination process performed in a target determination unit 109. 12 is a flowchart of a feature amount calculation storage process performed in a feature amount calculation storage unit according to the second embodiment. It is the figure which put together the example of the determination method of the tracking target performed in the target determination process part 109. FIG. It is the figure which put together the kind of feature-value, and the determination method of a tracking target. It is an example of a feature-value table.

Explanation of symbols

101: an imaging processing unit,
102: AD converter,
103: Pre-processing unit,
104: Feature enhancement unit,
105: binarization processing unit,
106: feature amount calculation storage unit,
107: Labeling processor
108: Consolidated investigation processing unit,
109: Target determination unit,
110: Tracking processing unit,
111: Gimbal mechanism.

Claims (4)

An AD conversion unit that converts the video signal obtained by the imaging processing unit into an image signal of a digital signal for each frame;
A labeling processing unit for giving a label to each object image in the image converted by the AD conversion unit;
An area calculation storage unit for storing an area of the object image to which the labeling processing unit has given a label and storing a feature amount table for storing the area for each label;
A connection investigation processing unit for detecting a positional relationship between the object image in the previous frame and the object image in the current frame;
The area calculation storage unit for each of the label correlation and the connection check operation unit detects the area of the each label of the object image in the object image and the current frame in the previous frame stored in the feature quantity table When it is determined from the positional relationship that the tracking target that existed in the previous frame has disappeared in the current frame and an object image has occurred within a certain distance from the position of the tracking target of the previous frame, among the generated object images and the object image and determines a target determination unit of the tracking target in the current frame object image the generated having an area with a high surface area correlation tracking target in the previous frame,
Based on the position of the tracking target determined by the target determination unit, a gimbal mechanism unit that controls the direction of the imaging processing unit,
A target detection tracking device. An AD conversion unit that converts the video signal obtained by the imaging processing unit into an image signal of a digital signal for each frame;
A labeling processing unit for giving a label to each object image in the image converted by the AD conversion unit;
A feature amount calculation storage unit that calculates a feature amount of the object image to which the labeling processing unit has given a label, and stores a feature amount table that stores the feature amount for each label;
A connection investigation processing unit for detecting a positional relationship between the object image in the previous frame and the object image in the current frame;
Correlation between feature quantities for each label of the object image in the previous frame and the object image in the current frame stored in the feature quantity table by the feature quantity calculation storage unit or for each label detected by the connection investigation processing unit When it is determined from the positional relationship that the tracking target that existed in the previous frame has disappeared in the current frame and an object image has occurred within a certain distance from the position of the tracking target of the previous frame, and the object image and determines a target determination unit of the tracking target in the current frame object image the generated with the characteristic quantity and high feature quantity correlation tracking target in the previous frame,
Based on the position of the tracking target determined by the target determination unit, a gimbal mechanism unit that controls the direction of the imaging processing unit,
A target detection tracking device.   The target detection tracking device according to claim 2, wherein the feature amount is any one of an area of the object image, a maximum luminance, a barycentric coordinate, and a moving direction.   The target detection tracking device according to claim 2, wherein the feature amount is a combination of any one of an area of an object image, maximum luminance, barycentric coordinates, and a moving direction.

Images