JP2021182267A - Tracking device - Google Patents

Abstract

To enable tracking an object to be tracked even when no feature points are obtainable.SOLUTION: A moving speed estimation device 10 includes: a feature amount extraction section 54 for, as to each image at each time when an object to be tracked is photographed, extracting feature amounts from each of multiple local areas; a tracking section 56 for, as to each of multiple local areas in an image, searching for local areas to which the feature amounts correspond, among the local areas in the image at a following time; an invalidation section 58 for, when a distance on the image to the searched local areas in the image at the following time as to the local areas in the image is equal to a threshold or greater, invalidating a searched result as to the local areas in the image; and a moving speed calculation section 60 for calculating a moving speed of the object to be tracked by averaging the moving speeds based on moving distances calculated from each correspondence of the local areas between the current image and the image at the following time.SELECTED DRAWING: Figure 2

Description

The present invention relates to a tracking device that tracks a tracking object.

Conventionally, there has been known a method in which a rolling line is photographed from above, the contour of a steel material contained in the image is captured, and tracking is performed according to the time change (for example, Patent Document 1).

Further, a method of detecting an edge line of a steel sheet and performing tracking is known (for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 62-104615 Japanese Patent Publication No. 2011-505257

However, with the methods described in Patent Documents 1 and 2, tracking cannot be performed because feature points cannot be obtained when the tail end is not in the field of view, such as when steel materials are connected.
Further, when a steel material on a hot rolling line is targeted, characteristic points may disappear or occur on the surface of the steel material to be imaged due to scale generation or dropout, but the above-mentioned Patent Documents 1 and 2 In the method described in the above, tracking cannot be performed unless feature points are obtained.

The present invention has been made in view of the above circumstances, and provides a tracking device capable of tracking a tracking object even when a feature point cannot be obtained.

In order to achieve the above object, the tracking device according to the first aspect of the present invention is a tracking device that tracks a tracking object, and the tracking target is for each of the images at each time when the tracking object is imaged. The target area including the area representing an object is divided into a plurality of local areas, and the feature amount extraction unit for extracting the feature amount from each of the plurality of local areas and each of the plurality of local areas of the image are as follows. It is configured to include a tracking unit for searching a local region corresponding to the feature amount from the local region of the image at the time of.
According to the tracking device according to the first aspect of the present invention, the target area including the region representing the tracking object is divided into a plurality of local regions, and the feature amount is extracted from each of the plurality of local regions. For each of the plurality of local regions of the image, the feature quantity is extracted even for the local region having no feature point by searching the local region corresponding to the feature quantity from the local region of the image at the next time. Then, since the corresponding local region is searched from the image at the next time, the tracking object can be tracked even when the feature point cannot be obtained. Further, since the target area is divided into a plurality of local areas and the search is performed in units of local areas, the amount of calculation required for the search can be suppressed.
Here, the feature point is a feature point in the image that can be distinguished from the surrounding points, and is, for example, an edge point or the like. The feature quantity is a quantity that represents the characteristics of the local region, which is calculated for the local region.

In the tracking device according to the second aspect of the present invention, when the distance on the image from the local region of the image at the next time searched by the tracking unit for the local region of the image is equal to or larger than the threshold value. It further includes an invalidation unit that invalidates the result of searching for the local area of the image.

The tracking device according to the third aspect of the present invention averages the moving speed based on the moving distance obtained from each of the correspondences of the local region between the image and the image at the next time after the invalidation by the invalidating unit. This further includes a movement speed calculation unit that calculates the movement speed of the tracking object.

In the tracking device according to the fourth aspect of the present invention, the feature amount extraction unit approximates the distribution of the luminance values of the pixels in the local region by an nth-order polynomial (n is an integer of 2 or more). The coefficient of the order polynomial is extracted as the feature quantity.

In the tracking device according to the fifth aspect of the present invention, the tracking unit has the feature amount on the assumption that the pixels in the local region and the pixels in the local region of the image at the next time move uniformly. Search for the corresponding local area.

In the tracking device according to the sixth aspect of the present invention, the tracking object is a steel material being conveyed on a hot rolling line.

In the tracking device according to the seventh aspect of the present invention, the hot rolling line is a continuous rolling line.

According to the present invention, the trackable object can be tracked even when the feature points cannot be obtained.

It is a figure which shows an example of the schematic structure of a hot rolling system. It is a block diagram of the movement speed estimation device. It is a figure which shows an example of the image for every frame. It is a figure for demonstrating the invalidation method of the tracked local area. It is a flowchart of a moving speed estimation process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the embodiment of the present invention, the case where the tracking device is applied to the movement speed estimation device for estimating the movement speed of the slab will be described as an example.

<Outline of Embodiment of the present invention>
Since the steel materials are connected by welding during transportation, tracking using the tail end as a mark is difficult.

In addition, since there is no identification target such as a marking or mark such as a steel piece No. on the steel material, it is difficult to track using them as a mark.

Therefore, in the embodiment of the present invention, the image of the steel piece conveyed on the continuous rolling line is photographed by a fixed camera, and the feature amount is extracted for each local region in the region representing the steel material for the image of each frame. , The feature quantity is used to search for the existence of the same local region in the next frame. Then, if there is the same local area, the difference between the coordinates in the image is calculated and calculated as the movement amount. At that time, those having a value that cannot exist as a movement amount are excluded. When a plurality of movement amounts are detected in the image, the running speed of the steel material is estimated using the average value and the median value.

<Outline of hot rolling system configuration>
FIG. 1 is a diagram showing an example of a schematic configuration of a hot rolling system, which is an example of an application destination of the moving speed estimation device 10.

In FIG. 1, the hot rolling system includes a heating furnace 11, a welding machine 12, a widthwise rolling mill 13, a rough rolling mill 14, a finishing rolling mill 15, a cooling device (runout table) 16, and winding. It has a device (coiler) 17.

The heating furnace 11 heats the slab (steel material) S.

The welding machine 12 is welded and connected while the slab S heated in the heating furnace 11 is being conveyed by using a traveling carriage (not shown).

The width direction rolling mill 13 rolls the slab S connected by the welding machine 12 in the width direction.

The rough rolling mill 14 rolls the slab S rolled in the width direction by the width rolling mill 13 from the vertical direction into a rough bar. In the example shown in FIG. 1, the rough rolling mill 14 has a rolling stand 14a made of only a work roll and rolling stands 14b to 14e having a work roll and a backup roll.

The finish rolling mill 15 continuously hot finish rolls the rough bar manufactured by the rough rolling mill 14 to a predetermined thickness. In the example shown in FIG. 1, the finishing rolling mill 15 has seven rolling stands 15a to 15g.

The cooling device 16 cools the hot-rolled steel sheet H (hereinafter, simply referred to as the steel sheet H) that has been hot-finished and rolled by the finish rolling mill 15 with cooling water.

The winding device 17 winds the steel plate H cooled by the cooling device 16 into a coil shape.

The hot rolling system can be realized by a known technique and is not limited to the configuration shown in FIG. 1.

Further, at the installation location of the welding machine 12, a fixed camera C is provided on the side surface side of the slab S. As a result, the fixed camera C captures the side surface of the slab S and outputs an image of each frame.

<Configuration of moving speed estimation device>
Next, the configuration of the moving speed estimation device will be described. FIG. 2 shows the functional configuration of the moving speed estimation device 10.

As shown in FIG. 2, the movement speed estimation device 10 includes an image acquisition unit 52, a feature amount extraction unit 54, a tracking unit 56, an invalidation unit 58, a movement speed estimation unit 60, and a display unit 62.

The fixed camera C is arranged so as to be perpendicular to the transport direction A of the slab S connected by welding by the welding machine 12 (not shown in FIG. 2) and to photograph the side surface of the slab S.

The fixed camera C captures an image (for example, a color image) representing the side surface of the slab S frame by frame.

The image acquisition unit 52 acquires an image taken by the fixed camera C for each frame.

As shown in FIG. 3, an image is acquired for each frame, and a target area including a region representing a side surface of the slab S is detected for each of the acquired images for each frame.

The feature amount extraction unit 54 divides the target area including the area representing the side surface of the slab S into a plurality of local areas for each of the images acquired for each frame by the image acquisition unit 52, and from all of the plurality of local areas. Each feature quantity is extracted.

Specifically, the feature amount extraction unit 54 uses the coefficient of the nth-order polynomial when the distribution of the luminance values of the pixels in the local region is approximated by the nth-order polynomial (n is an integer of 2 or more) as the feature amount. Extract.

_{For example, the coefficients A 1} , b ₁ , and c ₁ of the quadratic polynomial when approximated by the following two-dimensional polynomial are extracted as features.

（１）

(1)

However, f ₁ (x) represents the luminance value of the pixel x in the local region. Further, x is a two-dimensional vector representing the image coordinates of the pixels.

The tracking unit 56 sets each frame as a target frame based on the feature amount extracted from all of a plurality of local regions for each frame by the feature amount extraction unit 54, and for each of all the local regions of the image of the target frame. From the local area of the image of the next frame, the local area corresponding to the feature amount is searched.

Specifically, for each of all the local areas of the image of the target frame, the local area of the image of the next frame is compared with the feature amount of all the local areas of the image of the next frame. Search for a local area where the area and the feature amount correspond.

Here, assuming that the local region of the target frame is moved by d in the image of the next frame, the distribution of the brightness values of the pixels in the local region moved by d in the image of the next frame is distributed by a quadratic polynomial. The quadratic polynomial when approximated is expressed by the following equation.

（２）

(2)

However, d is a two-dimensional vector representing the amount of movement of each coordinate axis.

Therefore, in the present embodiment, for example, the feature amount A of the local region of the image of the next frame satisfying the relationship of the following equation with respect to _{the feature quantities A 1} , b ₁ , and c _{1 of the local region of the target frame.} Search for ₂ , b ₂ , and c _2.

（３）

(3)

However, d in the above equation is a two-dimensional vector representing the amount of movement of the local region of the image of the next frame with respect to the local region of the target frame.

When the moving distance on the image from the local area of the image of the next frame searched by the tracking unit 56 for the local area of the image of the target frame is equal to or larger than the threshold value, the invalidation unit 58 of the image of the target frame The result of searching for the local area is invalidated.

Specifically, for each of the local regions of the image of the target frame, when the local region corresponding to the feature amount is searched from the local regions of the image of the next frame, the moving direction of the local region is determined. It is determined whether or not the moving distance of the local region corresponds to the transport direction of the slab S and is less than a predetermined time of the moving distance based on the reference transport speed of the slab S, and the moving direction of the local region is the slab S. If it does not correspond to the transport direction of the local region, or if the travel distance of the local region is a predetermined time or more of the travel distance based on the reference transport speed of the slab S, the result of searching for the local region is invalidated.

For example, as shown in FIG. 4, the positive / negative of the difference (x ₁ − x _{2 ) between the x coordinate x 1 of the local region of the target frame and the x coordinate x 2} of the corresponding local region of the next frame is the slab S. When it does not correspond to the transport direction (for example, x ₁ − x ₂ > 0), it is determined that the movement direction of the local region does not correspond to the transport direction of the slab S. _{In addition, the moving distance (x 1} − x ₂ ) in the x-axis direction of the local region is determined by using a predetermined multiple (for example, 1.1 times) determined so as not to erroneously detect scale scattering on the slab S. , It is determined whether or not it is less than 1.1 times the moving distance based on the reference transport speed of the slab S.

The movement speed estimation unit 60 of the slab S by averaging the movement speeds based on the movement distances obtained from each of the correspondences of the local regions with the image of the next frame after the invalidation by the invalidation unit 58. Calculate the moving speed.

Specifically, for each of all the local areas of the image of the target frame, the local area corresponding to the feature amount is searched from the local area of the image of the next frame, and the invalidation unit 58 invalidates the local area. If not, the movement speed based on the movement distance of the local region obtained from each position of the corresponding local region is calculated, and the average of the calculated movement speeds is calculated as the movement speed of the slab S. Instead of averaging the moving speeds, the median value of the calculated moving speeds may be calculated as the moving speeds of the slab S.

The display unit 62 displays the estimation result of the moving speed estimation unit 60.

Next, the movement speed estimation process executed by the movement speed estimation device 10 will be described with reference to the flowchart shown in FIG. The process shown in FIG. 5 is repeatedly executed for each frame of the image captured by the fixed camera C while the slab S connected by the welding machine 12 is being conveyed in the imaging range of the fixed camera C. ..

In step S100, the image acquisition unit 52 acquires the image of the t-th frame and the image of the t + 1-th frame from the fixed camera C.

In step S102, the feature amount extraction unit 54 divides the target area including the area representing the side surface of the slab S into a plurality of local areas for each of the images of the t-th and t + 1-th frames acquired by the image acquisition unit 52. do.

In step S104, the feature amount is extracted from all of the plurality of local regions for each of the images of the t-th and t + 1-th frames.

In step S106, a local region of the image of the t-th frame is selected.

In step S108, the tracking unit 56 searches the local area of the image of the next frame for the selected local area of the image of the t-th frame, and searches for the local area corresponding to the feature amount, and selects the local area. The moving distance obtained from the position of the region and the position of the searched local region is calculated.

In step S110, it is determined whether or not the travel distance calculated in step S108 is valid. It is determined to be effective when the moving direction of the local region corresponds to the transport direction of the slab S and the moving distance of the local region is less than a predetermined time of the moving distance based on the reference transport speed of the slab S. Then, the process proceeds to step S112, and if not, it is determined that the process is invalid, and the process proceeds to step S116.

In step S112, the moving distance calculated in step S108 is drawn on the image of the t-th frame, and in step S114, the moving distance calculated in step S108 and the time corresponding to the frame interval are set. Based on this, the travel distance is calculated.

In step S116, it is determined whether or not the processing of steps S106 to S114 has been executed for all the local regions of the image of the t-th frame, and the local regions that have not executed the processing of steps S106 to S114 are If it exists, the process returns to step S106 and the local region is selected. On the other hand, when the processes of steps S106 to S114 are executed for all the local regions of the image of the t-th frame, the movement speed calculated in step S114 is averaged in step S118, and the display unit 62 Is displayed and the movement speed estimation process is completed.

As described above, in the present embodiment, the image of the steel piece conveyed on the continuous rolling line is captured by a fixed camera, and the feature amount is extracted for each local region in the region representing the steel material for the image of each frame. The feature quantity is used to search for the existence of the same local region in the next frame. Then, if there is the same local area, the difference between the coordinates in the image is calculated and calculated as the movement amount. At that time, those having a value that cannot exist as a movement amount are excluded. This makes it possible to track the slab being conveyed even when the feature points cannot be obtained, such as when the tip and tail ends of the slab being conveyed on the continuous rolling line are not in the field of view.

In addition, by extracting the coefficient when the distribution of the brightness value of the pixel in the local area is approximated by a quadratic polynomial as the feature amount, the local area can be compared with the case where the normal feature amount related to the brightness value etc. is used. It is possible to determine the exact degree of matching, and it is possible to accurately track the slab being transported.

Further, even when the pattern or the like is unknown, the local region can be traced because the local region corresponding to the feature amount is searched from all the local regions. At this time, even if an erroneous response is searched for, it can be invalidated according to the travel distance.

<Modification example>
In the above embodiment, a case where a coefficient of a two-dimensional polynomial that approximates the distribution of the luminance value of the image in the local region is extracted as a feature quantity, and a local region having the corresponding feature quantity is searched for and tracked is taken as an example. As described above, the present invention is not limited to this, and the corresponding local region may be searched by assuming that the local region of the image moves uniformly.

In this case, the feature amount extraction unit 54 extracts the luminance value of each pixel in the local region as the feature amount.

On the assumption that the pixels in the local region and the pixels in the local region of the image of the next frame move uniformly, the tracking unit 56 covers all the local regions of the image of the target frame by the method described below. For each, the local area corresponding to the feature amount is searched from the local area of the image of the next frame.

First, consider the following energy function.

（４）

(4)

Here, B represents a rectangular local region centered on the pixel x, and d is a parameter representing an arbitrary position of the local region. That is, it is the sum of squares of the luminance differences between the corresponding points in the local region before and after the movement of the pixel x when the movement from the t-th frame to the t + 1th frame is v _t. If all the pixels in the local region move in the same direction by the same amount with the same brightness, the above equation (4) becomes 0. By approximating the right-hand side of the above equation (4) with a first-order Taylor expansion, the following equation is obtained.

（５）

(5)

である。 However,

Is.

∇f _t represents a difference value in the horizontal direction and the vertical direction of the image f _t at pixel x, f _t denotes the time difference between adjacent frames at the pixel x. The movement amount v that minimizes the sum of the squared luminance differences is obtained for each pixel x.

By differentiating Eq. (5) with v _t and setting it to 0, the optimum amount of movement v _t can be obtained as shown in the following equation.

（６）

(6)

Since the movement amount in each pixel is not independent, the calculation process of the following formula is repeated _{until v t} ^{k + 1} (x) does not change in all pixels x to determine the movement amount.

（７）

(7)

Further, in the present embodiment, the case where the fixed camera C is installed at the installation location of the welding machine 12 has been described, but the installation location is not limited to this.

10 Moving speed estimation device 11 Heating furnace 12 Welding machine 13 Width rolling machine 14 Rough rolling machine 15 Finishing rolling machine 16 Cooling device 17 Winding device 52 Image acquisition unit 54 Feature quantity extraction unit 56 Tracking unit 58 Invalidation unit 60 Movement speed Estimating unit 62 Display unit C Fixed camera H Steel plate S Slab

Claims (7)

A tracking device that tracks objects to be tracked.
For each of the images at each time when the tracking object is imaged, the target area including the region representing the tracking object is divided into a plurality of local regions, and the feature amount is extracted from each of the plurality of local regions. Volume extractor and
For each of the plurality of local regions of the image, a tracking unit that searches for a local region corresponding to the feature amount from the local regions of the image at the next time.
Tracking device including. When the distance on the image from the local area of the image at the next time searched for the local area of the image by the tracking unit is equal to or more than the threshold value, the result of searching for the local area of the image is invalidated. The tracking device according to claim 1, further comprising an invalidation unit. The movement speed of the tracked object is determined by averaging the movement speeds based on the movement distances obtained from each of the correspondences of the local region with the image at the next time after invalidation by the invalidation unit. The tracking device according to claim 2, further including a moving speed calculation unit for calculation. The feature amount extraction unit extracts the coefficient of the nth-order polynomial when the distribution of the brightness values of the pixels in the local region is approximated by an nth-order polynomial (n is an integer of 2 or more) as the feature amount. The tracking device according to any one of claims 1 to 3. The tracking unit searches for a local region corresponding to the feature amount on the assumption that the pixel of the local region and the pixel of the local region of the image at the next time move uniformly. The tracking device according to any one of claims 3. The tracking device according to any one of claims 1 to 5, wherein the tracking object is a steel material being conveyed in a hot rolling line. The tracking device according to claim 6, wherein the hot rolling line is a continuous rolling line.

Images