JP6813244B2 - Airport surface monitoring device - Google Patents

Description

The present invention relates to an airport surface monitoring device that supports airport surface control work.

Conventionally, as an airport surface monitoring device that supports airport surface control work, position correlation and tracking are performed using the position of the target object detected by the airport surface detection radar and the position of the target object detected by the camera to perform the target. Some output information such as the position of an object (see, for example, Patent Document 1). This airport surface monitoring device uses image data of a video camera installed on the airport surface and character recognition by image processing to identify objects moving on the airport surface, and determines the position of the target object detected by the airport surface detection radar. It is designed to output by taking the correlation of.

JP-A-11-160424

However, in the above-mentioned conventional technique, the airport surface detection radar outputs the measurement error together with the position information, but the detection of the target object using the image obtained from the camera only outputs the position information and the measurement error. Is not output. Therefore, it is not possible to perform processing using a measurement error when performing position correlation and tracking, and therefore it is difficult to obtain high tracking accuracy.

The present invention has been made to solve the above problems, and an object of the present invention is to obtain an airport surface monitoring device that can contribute to improvement of tracking accuracy when tracking a target object.

The airport surface monitoring device according to the present invention includes a camera that acquires an image on the airport surface, a target detection unit that detects a target object from the image acquired by the camera and outputs image coordinates of the target object, and the target object is an airport. The 3D coordinates of the target object on the airport surface using the grounding position estimation unit that estimates the grounding position coordinates on the image that touches the surface, the estimation result of the grounding position coordinates, and the altitude information of the airport surface in the 3D coordinates. The measurement error at the position of the 3D coordinate of the target object is calculated by using the 3D position calculation unit that estimates the position in the above, the image coordinate of the target object, and the resolution data indicating the resolution of the 3D coordinate with respect to the image coordinate. It includes a measurement error estimation unit, an image coordinate of the target object, a position of the three-dimensional coordinate of the target object, and a result output unit that outputs the measurement error.

The airport surface monitoring device of the present invention uses a three-dimensional position calculation unit that estimates the position of a target object on the airport surface in three-dimensional coordinates, image coordinates of the target object, and resolution data, and uses the three-dimensional coordinates of the target object. Since it is equipped with a measurement error estimation unit that calculates the measurement error at the position of, it is possible to obtain an airport surface monitoring device that can contribute to the improvement of tracking accuracy when tracking a target object.

It is a block diagram which shows the airport surface monitoring apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the relationship between the airport surface to which the airport surface monitoring apparatus by Embodiment 1 of this invention is applied, and ENU coordinates. 3A and 3B are explanatory views of resolution data calculation of the airport surface monitoring device according to the first embodiment of the present invention. It is a flowchart which shows the resolution data calculation of the airport surface monitoring apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the target object detection of the N frame of the airport surface monitoring apparatus by Embodiment 1 of this invention. It is a block diagram which shows the airport surface monitoring apparatus by Embodiment 2 of this invention. It is a block diagram which shows the airport surface monitoring apparatus by Embodiment 3 of this invention. It is explanatory drawing which shows the estimation result of the target object grounding position of the N frame of the airport surface monitoring apparatus by Embodiment 3 of this invention. It is a block diagram which shows the airport surface monitoring apparatus according to Embodiment 4 of this invention. It is explanatory drawing which shows the estimation operation of the measurement error of the airport surface monitoring apparatus by Embodiment 4 of this invention.

Embodiment 1.
FIG. 1 is a configuration diagram showing an airport surface monitoring device according to the first embodiment of the present invention.
The airport surface monitoring device shown in FIG. 1 includes a camera 1, a target detection unit 2, a ground contact position estimation unit 3, a three-dimensional position calculation unit 4, a measurement error estimation unit 5, a result output unit 6, and a resolution calculation unit 7. The camera 1 is a device provided in one or more airports, which captures a moving path of a target object such as a runway or a taxiway and acquires an image on the airport surface. The target detection unit 2 is a processing unit that detects a target object such as an aircraft or a vehicle from the image acquired by the camera 1 and outputs the image coordinates of the target object. The ground contact position estimation unit 3 is a processing unit that estimates the image coordinates at which the target object and the airport surface are in contact with each other based on the image coordinates obtained by the target detection unit 2. The three-dimensional position calculation unit 4 is a processing unit that calculates three-dimensional coordinates indicating the three-dimensional position from the image coordinates obtained from the ground contact position estimation unit 3 where the target object and the airport surface are in contact with each other. The measurement error estimation unit 5 uses the resolution data 100 indicating the resolution of the three-dimensional coordinates with respect to the image coordinates and the image coordinates of the target object detected by the target detection unit 2, and the target output from the three-dimensional position calculation unit 4. This is a processing unit that calculates the measurement error of the three-dimensional coordinates of an object. The result output unit 6 is a processing unit that outputs the image coordinates of the target object, the three-dimensional coordinates of the target object, and the position measurement error as the monitoring result of the airport surface monitoring device. The resolution calculation unit 7 is a processing unit that calculates the resolution data 100 based on the calibration result of the three-dimensional coordinates.

Next, the operation of the airport surface monitoring device of the first embodiment will be described.
The image captured by the camera 1 is transferred to the target detection unit 2. The target detection unit 2 detects a target object such as a target aircraft or vehicle from images given in time series. For detection, a method using background subtraction, a method of extracting features from an aircraft in advance, and a method of detecting using learning data obtained by machine learning can be used. The detected target object is represented as a rectangle and transferred to the ground contact position estimation unit 3 and the measurement error estimation unit 5. The rectangle is represented by R = (u, v, w, h) using the upper left image coordinates, width and height. The ground contact position estimation unit 3 estimates the position of contact with the airport surface with respect to the target object detected by the target detection unit 2. In the first embodiment, the lower center (u + w / 2, v + h) of the detection rectangle is set as the ground contact position L. This grounding position L is transferred to the three-dimensional position calculation unit 4.

The three-dimensional position calculation unit 4 estimates the ENU (East, North, Up) coordinates using the ground contact position L estimated by the ground contact position estimation unit 3. As shown in FIG. 2, the ENU coordinate system sets the origin at an arbitrary location such as the position of the control tower 201 of the airport, and defines the eastward direction as the X-axis direction, the northward direction as the Y-axis direction, and the sky direction as the Z-axis direction. (See ENU coordinate system 202 in the figure). The ENU coordinates are transformed using the camera's internal parameters, the camera's rotation matrix, and the translation vector.
The three-dimensional position calculation unit 4 first converts the image coordinates to the camera coordinates.

例えば、Ｚ＝０のときのＸ、Ｙは、定数ｋを以下の式（４）（５）で求めることで、算出できる。

以上により、ＥＮＵ座標（Ｘ，Ｙ，Ｚ）を算出できる。

For example, X and Y when Z = 0 can be calculated by obtaining the constant k by the following equations (4) and (5).

From the above, the ENU coordinates (X, Y, Z) can be calculated.

As shown in FIG. 3, the resolution calculation unit 7 calculates the resolution in the ENU coordinates of the pixels as the resolution data 100. FIG. 3A shows the camera 301, the image coordinate system 302, and the ENU coordinate system 303. Further, FIG. 3B is an enlarged view within the broken line frame in the image coordinate system 302. The pixel resolution is obtained by calculating the Euclidean distance between the position before the shift and the position after the shift of the ENU coordinate system 303 when the pixel of interest 304 is shifted by 1 pixel in the u or v direction in the image coordinate system 302. can get. Since the direction of the deviation of 1 pixel is calculated along the axes of the X direction and the Y direction in the ENU coordinate system 303, it is calculated after reflecting the axial movement in the ENU coordinate system 303 in the image coordinate system 302. FIG. 4 shows the processing flow. Here, the points shown by ST1 to ST9 in FIG. 3 indicate the positions corresponding to the processes of steps ST1 to ST9 in FIG.

さらに、分解能算出部７は、式（７）に示すように、このＥＮＵ座標系３０３の位置を画像座標系３０２の位置に変換する（ステップＳＴ４）。

The resolution calculation unit 7 first selects the image coordinates (u, v) of the pixel of interest 304 (step ST1). Next, the resolution calculation unit 7 calculates the ENU coordinates (X, Y, Z) corresponding to the image coordinates (u, v) (step ST2). Note that this process may be performed by the three-dimensional position calculation unit 4. Next, the resolution calculation unit 7 calculates a point that is α-moved in the Y-axis direction in the ENU coordinate system 303 of the image coordinates (u, v) of the pixel of interest 304 as shown in the equation (6) (step ST3). ..

Further, the resolution calculation unit 7 converts the position of the ENU coordinate system 303 to the position of the image coordinate system 302 as shown in the equation (7) (step ST4).

として表され（ステップＳＴ５）、Ｙ軸方向への１ピクセル当たりの変化は、

次に、分解能算出部７は、画像座標系３０２の位置（ｕ，ｖ）と位置（ｕ_→ｙ，ｖ_→ｙ）とを、それぞれＥＮＵ座標系３０３の位置（Ｘ’，Ｙ’，Ｚ’）に変換し（ステップＳＴ８）、式（１１）に示すように、そのユークリッド距離を求める（ステップＳＴ９）ことでＹ軸のプラス方向へのピクセルの分解能を算出する。

As shown in FIG. 3B, the moving direction of the ENU coordinate system 303 in the image coordinate system 302 in the Y-axis direction is

(Step ST5), the change per pixel in the Y-axis direction is

Next, the resolution calculation unit 7 sets the position (u, v) and the position (u _{→ y} , v _{→ y} ) of the image coordinate system 302 to the position (X', Y', Z'of the ENU coordinate system 303, respectively. ) (Step ST8), and as shown in the equation (11), the Euclidean distance is obtained (step ST9) to calculate the pixel resolution in the positive direction of the Y axis.

After obtaining the Euclidean distance in step ST9, the resolution calculation unit 7 selects the next image coordinates (u, v) (step ST10) and returns to step ST2. The resolution calculation unit 7 repeats the process from step ST2 to calculate the resolution in the Y-axis direction.
Further, the resolution calculation unit 7 calculates the resolution of the pixels in the negative direction of the Y-axis by obtaining the Euclidean distance when Y−α is set in the equation (6), and calculates the average thereof in the Y-axis direction. Let it be the resolution.
The coordinates moved by 1 pixel in the X-axis direction can be calculated in the same manner. The resolution obtained by the above calculation is expressed as Q (u, v, offset = (X, Y)), and when there is a pixel shift in the (X or Y) axis direction in the image coordinates (u, v). Is the value of. This value is saved as the resolution data 100 in the calibration result.

次に、計測誤差推定部５は、Ｘ軸方向とＹ軸方向の計測誤差を算出する。現フレームの接地位置Ｌにおける分解能Ｑを抽出し、Ｘ、Ｙ軸方向へのベクトルと式（１２）、（１３）で算出した標準偏差を掛け合わせることにより計測誤差Ｓ_Ｘ、Ｓ_Ｙとする。

計測誤差推定部５は、結果出力部６に、検出矩形の情報とＥＮＵ座標と計測誤差Ｓ_Ｘ、Ｓ_Ｙを転送する。結果出力部６では、目標物体の画像座標とＥＮＵ座標と計測誤差Ｓ_Ｘ、Ｓ_Ｙとを空港面監視装置の監視結果として出力する。 The measurement error estimation unit 5 estimates the estimated error of the ENU coordinates by using the resolution data 100 and the information of the time-series detection rectangle from the target detection unit 2. FIG. 5 is an explanatory diagram showing a time-series detection rectangle from the target detection unit 2. First, as shown in the figure, the detection rectangle Ri (i = 0, ... N-1) of the target object in the N frame is used, and the standard deviations (S _w , _Sh ) of the width and height are calculated. In FIG. 5, the detection rectangle 51 is the detection frame at time t-3, the detection rectangle 52 is the detection frame at time t-2, the detection rectangle 53 is the detection frame at time t-1, and the detection rectangle 54 is the time t. The detection frame of is shown, and the set 55 shows a set of detection frames of 4 frames.

Next, the measurement error estimation unit 5 calculates the measurement error in the X-axis direction and the Y-axis direction. Extract the resolution Q at the ground position L of the current frame, X, vector and expression of the Y-axis direction (12), the measurement error S _X, S _Y by multiplying the standard deviation calculated in (13).

Measurement error estimating unit 5, the result output unit 6, and transfers the detection rectangle information and ENU coordinate measurement error S _X, the S _Y. The result output unit 6, and outputs the image coordinates and the ENU coordinate of the target object measurement errors S _{X, and} S Y as a monitoring result of the airport surface monitoring device.

As described above, according to the airport surface monitoring device of the first embodiment, the camera that acquires the image on the airport surface, the target object is detected from the image acquired by the camera, and the image coordinates of the target object are output. Using the target detection unit, the grounding position estimation unit that estimates the grounding position coordinates on the image where the target object touches the airport surface, the estimation result of the grounding position coordinates, and the altitude information of the airport surface in three-dimensional coordinates, the airport Three-dimensional of the target object using the three-dimensional position calculation unit that estimates the position of the target object on the surface in three-dimensional coordinates, the image coordinates of the target object, and the resolution data indicating the resolution of the three-dimensional coordinates with respect to the image coordinates. Since it is equipped with a measurement error estimation unit that calculates the measurement error at the coordinate position, an image coordinate of the target object, a position of the three-dimensional coordinate of the target object, and a result output unit that outputs the measurement error, the target object It is possible to obtain an airport surface monitoring device that can contribute to the improvement of tracking accuracy when performing tracking.

Further, according to the airport surface monitoring device of the first embodiment, the three-dimensional position calculation unit estimates the position of the target object on the airport surface in the three-dimensional coordinates with the altitude information of the airport surface as a constant value. Therefore, it is possible to speed up the arithmetic processing.

Embodiment 2.
In the second embodiment, the resolution is calculated using the altitude information of the airport surface. That is, in the first embodiment, the resolution calculation unit 7 calculates the resolution of the three-dimensional coordinates with the altitude of the airport surface as a constant value, but in the second embodiment, the resolution is calculated using the actual altitude information of the airport surface. Is calculated.

FIG. 6 is a configuration diagram showing the airport surface monitoring device of the second embodiment.
The airport surface monitoring device shown in FIG. 6 includes a camera 1, a target detection unit 2, a ground contact position estimation unit 3, a three-dimensional position calculation unit 4, a measurement error estimation unit 5, a result output unit 6, and a resolution calculation unit 7a. Here, the difference from the airport surface monitoring device of the first embodiment is that the three-dimensional position calculation unit 4 is configured to calculate the three-dimensional coordinates using the altitude information data 101 indicating the altitude of the airport surface. The point is that the resolution calculation unit 7a calculates the resolution data 100a using the altitude information data 101. Here, the altitude information data 101 is data indicating altitude information of each position on the airport surface, that is, a value in the Z direction. Since the other configurations are the same as those shown in FIG. 1, the corresponding parts are designated by the same reference numerals and the description thereof will be omitted.

Next, the operation of the airport surface monitoring device of the second embodiment will be described. The points that are the same as the operation of the first embodiment will be omitted, and the operation different from that of the first embodiment will be described.
In the three-dimensional position calculation unit 4 of the second embodiment, the ENU coordinates are estimated using the ground contact position L estimated by the ground contact position estimation unit 3, as in the first embodiment, but the ENU coordinates are estimated. Advanced information data 101 is used for. By using the altitude information data 101, the equations (4) and (5) in the first embodiment become the following equations (16) and (17), and the ENU coordinates including the altitude information can be estimated. ..

Further, the resolution calculation unit 7a calculates the resolution data 100a using the altitude information data 101. The basic operation is the same as the operation using FIGS. 3 and 4 in the first embodiment, except that the calculation including the resolution in the Z-axis direction is included.

As described above, according to the airport surface monitoring device of the second embodiment, the resolution data is calculated by using the altitude of each position on the airport surface as the altitude information of the airport surface and using this altitude information. Therefore, the calculation accuracy of the resolution is improved, and the calculation accuracy of the measurement error of the three-dimensional coordinates can be improved.

Embodiment 3.
In the third embodiment, the ground contact position estimation unit estimates the ground contact position coordinates using the ground contact position learning data indicating the learning result of the feature amount of the target object and the ground contact position.
FIG. 7 is a configuration diagram showing the airport surface monitoring device of the third embodiment.
The airport surface monitoring device shown in FIG. 7 includes a camera 1, a target detection unit 2, a ground contact position estimation unit 3a, a three-dimensional position calculation unit 4, a measurement error estimation unit 5, a result output unit 6, and a resolution calculation unit 7a. Here, the difference from the airport surface monitoring device of the second embodiment is that the ground contact position estimation unit 3a uses the ground contact position learning data 102 indicating the learning result of the feature amount of the target object and the ground contact position to obtain the ground contact position coordinates. It is a point that is configured to be estimated. Since the other configurations are the same as those of the second embodiment shown in FIG. 6, the corresponding parts are designated by the same reference numerals and the description thereof will be omitted.

As a method of generating the grounding position learning data 102, for example, it is shown in the non-patent document Hough Forests (J. Gall, and V. Lempitzky, “Class-special forests for objects”, CVPR2009.). There is a method of extracting a finer NxN patch from a rectangular area in which an object is specified and finding the relationship between the patch and the vector to the grounding position. Then, in the grounding position estimation unit 3a, when estimating the grounding position, a patch is extracted from the rectangle obtained by the target detection unit 2, and Houch Forests is used to calculate a vector to the grounding position for each patch. it can. Let the grounding position [p, q] ^T be the maximum point in voting for the grounding position of all patches.

The measurement error estimation unit 5 estimates the error of the ENU coordinates estimated by the three-dimensional position calculation unit 4 by using the resolution data 100a and the information of the time-series detection rectangle from the target detection unit 2. Here, the measurement error estimation unit 5 of the first embodiment uses the standard deviation of the width and height of the detection frame, but the measurement error estimation unit 5 of the third embodiment is grounded as shown in FIG. position _(p i, _{q i)} and coordinates of the detection frame _(u i, _{v i)} using, as shown in equation (18) (19), the standard deviation of the coordinate value based on the upper left coordinates of the detection frame Calculate ( _Sp , S _q ). In FIG. 8, the detection rectangles 51 to 54 show the same detection frame as in FIG. 5, and the estimated grounding positions 81 to 84 are the estimated grounding position at time t-3 and the estimated grounding position at time t-2, respectively. , The estimated grounding position at time t-1 and the estimated grounding position at time t are shown. Further, the set 85 shows a set of grounding positions with reference to the upper left of the detection frame of 4 frames. Here, the arrows from the circles shown in the set 85 indicate the vector in the u-coordinate direction and the vector in the v-coordinate direction at the estimated grounding positions 81 to 84, respectively.

Next, the measurement error estimation unit 5 calculates the measurement error using the equations (14) and (15) as in the first embodiment. However, equation (14) _S w in (15), the _{S h} of the _S p, _{S q} as measurement errors _S X, obtaining the _{S Y.} Since the subsequent operations are the same as those in the first and second embodiments, the description thereof is omitted here.

In the above example, the configuration in which the grounding position estimation unit 3a estimates the grounding position using the grounding position learning data 102 has been applied to the second embodiment, but it is applied to the configuration of the first embodiment. You may.

As described above, according to the airport surface monitoring device of the third embodiment, the ground contact position estimation unit uses the ground contact position learning data indicating the learning result of the feature amount of the target object and the ground contact position to determine the ground contact position coordinates. Since the estimation is performed, the estimation accuracy of the ground contact position used when calculating the three-dimensional position can be improved, and as a result, the calculation accuracy of the three-dimensional position and the measurement error can be improved.

Embodiment 4.
In the fourth embodiment, the measurement error estimation unit estimates the measurement error using the three-dimensional coordinates.
FIG. 9 is a configuration diagram showing the airport surface monitoring device of the fourth embodiment.
The airport surface monitoring device shown in FIG. 9 includes a camera 1, a target detection unit 2, a ground contact position estimation unit 3, a three-dimensional position calculation unit 4, a measurement error estimation unit 5a, a result output unit 6, and a resolution calculation unit 7. Here, the difference from the airport surface monitoring device of the first embodiment is that the measurement error estimation unit 5a measures the distance between the straight line that approximates the three-dimensional coordinates of the time series of the target object and the three-dimensional coordinates of the time series. This is a point that is configured to be output as an error. Since the other configurations are the same as those of the first embodiment shown in FIG. 1, the corresponding parts are designated by the same reference numerals and the description thereof will be omitted.

であり、係数ａ，ｂは以下の式（２１）（２２）で得られる。

Next, as the operation of the fourth embodiment, the measurement error estimation process by the measurement error estimation unit 5a will be described.
As the measurement error, as shown in FIG. 10, a straight line is fitted using the ENU coordinates of a plurality of frames, and the deviation from the straight line is used as the measurement error. In FIG. 10, positions 11 to 14 indicate the ENU coordinates at time t-3, the ENU coordinates at time t-2, the ENU coordinates at time t-1, and the ENU coordinates at time t, respectively. Further, the straight line 15 indicates a straight line fitted to these positions 11 to 14.
Assuming that the three-dimensional coordinates of the time series are the ENU coordinates (X _i , Y _i , Z _i ) (i = 0, ... N-1) of the N frame, the straight line 15 is obtained by the least squares method. However, since it is assumed that the target object is traveling on the airport surface, Z is assumed to be a constant value, and a straight line is considered in two dimensions of the (X, Y) coordinates. The straight line formula is

The coefficients a and b are obtained by the following equations (21) and (22).

となる。従って、計測誤差は以下の式（２４）（２５）で求められる。

Next, the measurement error estimation unit 5a calculates the error of each ENU coordinate with respect to this straight line. If each error is (EX _i , EY _i ),

Will be. Therefore, the measurement error is calculated by the following equations (24) and (25).

Since other operations are the same as those in the first embodiment, the description thereof is omitted here.
In the above example, the case where the configuration of the measurement error estimation unit 5a is applied to the first embodiment has been described, but it may be applied to the configuration of the second embodiment or the third embodiment.

As described above, according to the airport surface monitoring device of the fourth embodiment, the measurement error estimation unit determines the distance between the straight line that approximates the three-dimensional coordinates of the time series of the target object and the three-dimensional coordinates of the time series. Since it is output as a measurement error, the measurement error can be directly obtained in the ENU coordinate system, and the calculation accuracy of the three-dimensional coordinates of the target object can be improved.

In the present invention, within the scope of the invention, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment. ..

1 camera, 2 target detection unit, 3,3a grounding position estimation unit, 43D position calculation unit, 5,5a measurement error estimation unit, 6 result output unit, 7,7a resolution calculation unit, 100,100a resolution data, 101 Advanced information data, 102 grounding position learning data.

Claims (5)

With a camera that captures images on the airport surface,
A target detection unit that detects a target object from the image acquired by the camera and outputs the image coordinates of the target object.
A grounding position estimation unit that estimates the grounding position coordinates on the image where the target object touches the airport surface,
A three-dimensional position calculation unit that estimates the position of the target object on the airport surface in three-dimensional coordinates by using the estimation result of the ground contact position coordinates and the altitude information of the airport surface in three-dimensional coordinates.
A measurement error estimation unit that calculates a measurement error at the position of the three-dimensional coordinates of the target object using the image coordinates of the target object and resolution data indicating the resolution of the three-dimensional coordinates with respect to the image coordinates.
An airport surface monitoring device including an image coordinate of the target object, a position of a three-dimensional coordinate of the target object, and a result output unit for outputting the measurement error. The airport according to claim 1, wherein the three-dimensional position calculation unit estimates the position of the target object on the airport surface in three-dimensional coordinates by using the altitude information of the airport surface as a constant value. Surface monitoring device. The airport surface monitoring device according to claim 1, wherein the resolution data is calculated by using the altitude information of each position on the airport surface as the altitude information of the airport surface. The ground contact position estimation unit according to any one of claims 1 to 3, wherein the ground contact position estimation unit estimates the ground contact position coordinates by using the ground contact position learning data indicating the learning result of the feature amount of the target object and the ground contact position. The airport surface monitoring device according to any one of the items. Claims 1 to 4 are characterized in that the measurement error estimation unit outputs a distance between a straight line that approximates the three-dimensional coordinates of the time series of the target object and the three-dimensional coordinates of the time series as a measurement error. The airport surface monitoring device according to any one of the above.

Images