JP6925603B2 - Attitude control system for moving objects - Google Patents

Description

The present invention relates to an attitude control system for a moving body. More specifically, the present invention relates to an independent attitude control system for a small unmanned aerial vehicle (drone) capable of moving in three-dimensional space as a moving body.

In recent years, small unmanned aerial vehicles, typically drones, have been used in a wide range of fields. In general, a drone captures radio waves from a GPS satellite with a GPS sensor mounted on the drone to calculate position data, and then uses an IMU (inertial measurement unit) mounted on the drone to perform movement (speed, acceleration, etc.) and posture (speed, acceleration, etc.) and posture ( (Inclination, etc.) data is calculated, and these position data and flight data are transmitted to the user's control device. The user controls the operation by transmitting flight control commands such as a movement command and an attitude control command to the drone while referring to these data.

Regarding the flight control of the drone, there is a conventional method of installing a reflection marker on the ground, mounting an infrared high-speed camera on the drone, and using a motion capture system. There are also systems that use monocular cameras, stereoscopic cameras, wide-range measuring instruments, etc. instead of infrared high-speed cameras.

The present inventor is aware that the following non-patent documents exist as prior art documents regarding attitude control of small unmanned aerial vehicles.

D. Mellinger and V. Kumar, “Minimum snap trajectory generation and control for quadrotors,” in IEEE International Conference on Robotics and Automation, May 2011, pp.2520-2525 M. Hehn and R. D Andrea, Quadcopter trajectory and control, in Proceedings of the IFAC World Congress, 2011 Michael, Nathan, et al. “The GRASP multiple micro-UAV testbed.” Robotics Automation Magazine, IEEE 17.3 (2010): 56-65 Heng, Lionel; Meiser, L .; Tanskanen, P .; Fraundorfer, F .; Pollefeys, M., “Autonomous obstacle avoidance and maneuvering on a vision-guided MAV using on-board processing,” in Robotics and Automation (ICRA) , 2011 IEEE International Conference on, vol., no., Pp.2472-2477, 9-13 May 2011 Stephan Weiss, Markus W. Achtelik, Simon Lynen, Margarita Chli, Roland Siegwart, “Real-time onboard visual-inertial state estimation and self-calibration on MAVs in unknown environments.” Robotics and Automation (ICRA), 2012 IEEE International Conference pp .957, May 2012 Shaojie shen; Michael, N .; Kumar, V., “Tightly-coupled monocular visual-inertial fusion for autonomous flight of rotorcraft MAVs,” in Robotics and Automation (ICRA), 2015 IEEE International Conference on, vol., No., pp.5303-5310, 26-30 May 2015 S. Grzonka, G. Grisetti, and W. Burgard, “Towards a navigation system for autonomous indoor flying,” in IEEE International Conference on Robotics and Automation, 2009 G. Xu, Y. Zhang, S. Ji, Y. Cheng, Y. Tian, Research on computer vision-based for UAV autonomous landing on a ship, Pattern Recognition Letters, Vol. 30 (6), 2009, pp.600 -605 Lange, S .; Sunderhauf, N .; Protzel, P., “A vision based onboard approach for landing and position control of an autonomouse multirotor UAV in GPS-defined environments” in Advanced Robotics, 2009. ICAR 2009. International Conference on, vol., pp.1-6, 22-26 June 2009 Lugo, Jacobo Jimenez, and Andres Zell, “Framework for autonomous on-board navigation with the AR. Drone.” Journal of Intelligent and Robotic Systems 73.1-4 (2014): 401-412 Troiani, Chiara, Stefano Al Zanati, and Alessio Martineli. “A 3 points vision based approach for mav localization in GPS defined environments.” Mobile Robots (ECMR), 2013 European Conference on. IEEE, 2013. Vogt, Sebastian, et al. “Single came tracking of marker clusters: Multiparameter cluster optimization and experimental verification” Proceedings of the 1st international Symposium on Mixed and Augmented Reality. IEEE Computer Society, 2002. Nikolic, Janosch, et al. “A synchronized visual-inertial sensor system with FPGA pre-processing for accurate real-time SLAM.” Robotics and Automation (ICRA), 2014 IEEE International Conference on. IEEE, 2014 Zhou, Guyue, et al. “On-board inertia-assisted visual odometer on an embedded system.” Robotics and Automation (ICRA), 2014 IEEE International Conference on. IEEE, 2044. Chen, Qian, Haiyuan Wu, and Toshikazu Wada. “Camera calibration with two arbitrary coplanar circles.” Computer Vision-ECVV 2004. Springer Berlin Heidelberg, 2004. 521-532 Elqursh, Ali, and Ahmed Elgammal. “Line-based relative pose estimation.” Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEEE, 2011 Zhang, Zhengyou. “A flexible new technique foe camera calibration.” Pattern Analysis and Machine Intelligence, IEEE Transaction on 22.11 (2000): 1330-1334 E. Rosten, R. Porter, and T. Drummond. “Faster and better: A machine learning approach to corner detection.” IEEE Trans. Pattern Analysis and Machine intelligence, 32: 105119, 2010 Harris, Chris, and Mike Stephens. “A combined corner and edge detector.” Alvey vision conference. Vol.15. 1988 Konomura, Ryo, and Koichi Hori. “Phoenix: Zynq 7000 based quadcopter robot. ReConFigurable Computing and FPGAs (ReConFig), 2014 International Conference on. IEEE, 2014.

The attitude control system proposed conventionally has a problem that the configuration is relatively complicated like a motion capture system using a reflection marker and an infrared high-speed camera, and an error of attitude control is relatively large. In addition, the attitude control hardware mounted on a small unmanned aerial vehicle has strict weight restrictions, and weight reduction is required.

Therefore, the present inventor proposes an attitude control system for a small unmanned aerial vehicle having a new and simple configuration different from the conventional attitude control method.

Specifically, an object of the present invention is to provide an independent attitude control system for a moving body (for example, a small unmanned aerial vehicle) having a novel and simple configuration.

Furthermore, it is an object of the present invention to provide a marker with a novel and simple configuration suitable for use in a self-sustaining attitude control system for a moving body (eg, a small unmanned aerial vehicle).

In one aspect of the invention, the attitude control system for a moving body according to the present invention includes a monocular camera and an attitude control device mounted on the moving body, and a three-dimensional marker installed at a predetermined location. Controls the independent attitude of the moving body based on the image data of the three-dimensional marker captured by the monocular camera.

Further, in the attitude control system of the moving body, the moving body may be an unmanned small airplane.

Further, in the movement control system of the moving body, the three-dimensional marker may have a shape having at least one plane and another plane or point not included in the plane.

Further, in the moving body posture control system, the posture control device is known in advance regarding image data of one plane of the three-dimensional marker captured by the monocular camera and other planes or points not included in the plane. Three-dimensional posture control may be performed by converting the coordinates based on the difference between the position being measured and the position at the time of measurement.

Further, in the moving body posture control system, the posture control device acquires an image of a three-dimensional marker from the monocular camera, converts the acquired image into a scalar image, and acquires the contour of a color object from the scalar image. Each vertex is detected from the contour of the color object at the 3D marker and subpixel level and sent to the marker management unit, the initial posture estimation is performed using each vertex of the plane, and interactive through bundle adjustment using each vertex of the 3D. The posture estimation process may be performed.

Further, in the motion control system of the moving body, the three-dimensional marker may be a marker having any shape of a cone, a triangular pyramid, a quadrangular pyramid, or another polygonal pyramid.

Further, in the moving body posture control system, the three-dimensional marker may have a quadrangular pyramid shape, and the marker may be asymmetrically colored by changing the color of at least one of the four triangular surfaces on the surface.

Further, in the posture control system of the moving body, the three-dimensional marker may be a marker arranged so as to represent a three-dimensional shape with a plurality of luminescent materials.

Further, in the attitude control system of the moving body, the moving body may be an industrial robot.

Further, in the attitude control system of the moving body, the moving body may be a truck for transferring articles traveling on a straight rail.

Further, the marker according to the present invention is, in one aspect of the invention, a three-dimensional marker used in an independent posture control system for a moving body that can move three-dimensionally.

Further, the three-dimensional marker is a three-dimensional marker having a shape having at least one plane and another plane or point not included in the plane.

According to the present invention, it is possible to provide an independent attitude control system for a moving body (for example, a small unmanned aerial vehicle) having a novel and simple configuration.

Further, according to the present invention, it is possible to provide a marker having a novel and simple configuration suitable for use in an independent attitude control system of a moving body (for example, a small unmanned aerial vehicle).

FIG. 1A is a diagram illustrating an image of an independent attitude control system for a drone according to the present embodiment. FIG. 1B is a diagram illustrating 6-Dof (6 degrees of freedom) attitude control of the drone. FIG. 2A is a block diagram of an attitude control device mounted on the drone in the independent attitude control system of the drone shown in FIG. FIG. 2B is a diagram showing an example of a marker used in the independent attitude control system of the drone shown in FIG. FIG. 3 is a diagram showing a flow of 6-Dof posture measurement according to the present embodiment. FIG. 4A is a diagram showing 16 points on the marker when the _{2D shape marker makes a small movement t x} in order to compare the 2D shape marker and the 3D shape marker. FIG. 4B is a diagram showing 16 points on the marker when the _{3D shape marker makes a small movement t x} in order to compare the 2D shape marker and the 3D shape marker. FIG. 4C is a diagram showing 16 points on the marker when the _{two-dimensional shape marker makes a small rotation θ y} in order to compare the two-dimensional shape marker and the three-dimensional shape marker. FIG. 4D is a diagram showing 16 points on the marker when the _{3D shape marker makes a small rotation θ y} in order to compare the 2D shape marker and the 3D shape marker.

Hereinafter, embodiments of the attitude control system for a moving body according to the present invention will be described with reference to the accompanying drawings, taking as an example a small unmanned aerial vehicle (drone) capable of moving in three-dimensional space as a moving body. In the drawings, the same reference numerals indicate the same elements, and duplicate description will be omitted.

[Image of drone attitude control]
FIG. 1A is a diagram illustrating an image of an independent attitude control system for a drone according to the present embodiment. The feature of this embodiment is that the independent attitude control system of the drone has a simple configuration. That is, the drone 2 autonomously controls the attitude while photographing the marker 6 installed on the ground in advance with the camera 20 mounted on the drone 2.

For self-sustaining attitude control, FIG. 1A has a simple configuration that does not require data from the GPS satellite 1 of the element shown by the broken line, data from the control device 4 of the user 3, and the like.

FIG. 1B is a diagram illustrating 6-Dof (6 degrees of freedom) attitude control of the drone. Generally, the degree of freedom of movement that an object can take in three-dimensional space is called 6-Dof (6 degrees of freedom), and specifically, the object can move forward or backward, up or down, left or right (in other words). For example, it means that it can move along each of the three-dimensional orthogonal coordinate axes), and that it can rotate around each axis of the orthogonal coordinate axes. For this rotation, aircraft such as Drone 2 use pitching, yawing, and rolling. That is, pitching means rotating around the left and right sides of the drone (so-called up and down). Yawing refers to rotation (in the so-called horizontal plane) about the top and bottom of the drone. Rolling refers to rotating (or tilting) with respect to the front and rear axes of the drone.

[Independent attitude control system]
(hardware)
FIG. 2A is a block diagram of an attitude control device 8 mounted on the drone 2 in the independent attitude control system of the drone shown in FIG. 1A. The attitude control device 8 includes a monocular camera 20 mounted on a substrate 10, a CPU 16, and a memory 18. These may be composed of FPGA.

The triaxial gyro sensor 14 is mounted for verifying the effect of the present embodiment, and is not an essential element for the present embodiment. However, if desired, a triaxial gyro sensor 14 may be mounted.

Conventionally, an accelerometer has been indispensable for measuring the rolling angle and pitching angle of the drone 2 with respect to the direction of gravity. However, in the 6-Dof attitude measurement of the present embodiment, it is possible to control the flight of the drone in a stable flight attitude and position without providing the attitude control device 8 with an accelerometer.

The monocular (single-lens) camera 16 is a one-lens camera. Usually, a binocular (twin-lens) camera using two lenses is required because parallax is used for distance measurement and 3D photography. In this embodiment, the posture can be controlled by the monocular camera by devising the marker and the posture measurement algorithm described later.

The memory 14 has a storage area for writing the attitude control algorithm and a work area for arithmetic processing of the CPU 12.

The drone 2 is also equipped with a motion capture device in order to obtain reference data when measuring the effect of the posture measurement of this embodiment.

(Marker shape, design and installation location)
The marker 6 used in this embodiment is a "three-dimensional marker". In the present application documents, a "stereoscopic marker" means a shape having at least one plane and another plane or point not included in the plane. Typically, it is a three-dimensional marker.

Preferably, it is a quadrangular pyramid marker 6 as shown in FIG. 2B. The quadrangular pyramid marker 6 is a combination of planes 6a, 6b, 6c, 6d, planes 6a, 6b, 6e, planes 6b, 6c, 6e, planes 6c, 6d, 6e, and planes 6d, 6a, 6e. be. From a different point of view, the quadrangular pyramid marker 6 has a shape in which the planes 6a, 6b, 6c, 6d and the point 6e are combined.
However, it is not limited to a quadrangular pyramid. It may be a cone or a polygonal pyramid.

Further, a non-uniform plane may be represented by a light emitting material such as an LED.

The number of markers is not limited to one. There may be two or more. Therefore, two plane markers may be installed on a plane that is not the same plane, and the two plane markers may be used as a "stereoscopic marker" as a whole.

With a quadrangular pyramid marker, the orientation in the yaw rotation direction can be distinguished by changing the color of at least one of the four triangular surfaces of the surface to make the marker asymmetric.

The location of the marker may be on the ground, in the air, or the like. If the position is clear in real time, it may move on the vehicle in chronological order. It can be installed on the car and the drone can follow it.

In the process of developing the marker, the planar shape (two-dimensional shape) and the three-dimensional shape (three-dimensional shape) were compared and examined regarding the shape of the marker. Finally, the shape of the marker was a three-dimensional shape with little estimation error of attitude control, preferably a quadrangular pyramid. The reason will be explained later.

(6-Dof algorithm)
The contents of this 6-Dof algorithm are, for example, the positions known in advance and the positions at the time of measurement for the five vertices (4 points of the base square and 1 point of the vertices) that identify the marker of the quadrangular pyramid. It is to digitize based on the difference of. Here, even in the single-lens (single-lens) camera 16 that captures two dimensions, three-dimensional attitude control is possible by the behavior of the vertices within the four points of the square of the quadrangular pyramid.

First, the posture symbols and methods used in this embodiment will be disclosed. The vector u _i = [u _i , v _i ] ^T is the i-th position of the image coordinates (0 ≤ u _i <H and 0 ≤ v _{i ≤ V).} Xi ∈ R ³ is the world coordinate. T = [t _x , t _y , t _z ] ^T ∈ R ³ and R ∈ SO (3) are relative camera movements and rotations. T _z is defined as the direction of the first camera, and θ = [θ _x , θ _y , θ _z ] is defined as the ZXY Euler angles of R. _{Next, the L 2} criterion for the reprojection error of a single image scene can be defined as follows.

In the process of developing the marker, the planar shape (two-dimensional shape) and the three-dimensional shape (three-dimensional shape) were compared and examined regarding the shape of the marker. Finally, the shape of the marker was a "three-dimensional shape" with little estimation error of attitude control, preferably a quadrangular pyramid. The reason will be briefly explained.

A simple example is shown to explain the difference between the coplanar marker (two-dimensional shape marker) and the non-coplanar marker (three-dimensional shape marker) taken by the camera. 3A-3D show 16 points of the marker when the camera is moved in two ways, with a small movement t _x (FIGS. 3A and 3B) and a small rotation θ _{y (FIGS. 3C and 3D).} With 2D shape markers, there is only a slight difference between translational and rotational movements. However, since the three-dimensional shape marker has one point (⊚) whose depth is different from that of other 15 points on the same plane, the difference in the changes in the _{movement t x} and the rotation θ _{y can be clearly seen.}

(6-Dof posture measurement flow)
FIG. 3 is a diagram showing a flow of 6-Dof posture measurement according to the present embodiment. This procedure is processed by the CPU 12 according to an algorithm written in the memory 14 in advance.

In step S01, an image including the marker 16 is acquired from the monocular camera 16.

In step S02, the acquired image is converted into a scalar image.

In step S03, the contour of the color object is obtained from the scalar image and sent to the CPU 12.

In step S04, the CPU 12 detects the vertices 6a to 6e at the marker 6 and the sub-pixel level from the contour of the color object.

In step S05, the marker 6 and the vertices 6a to 6e are sent to the marker management unit.

In step S06, the initial posture estimation is performed using the vertices 6a to 6d of the plane of the marker 6.

In step S07, interactive posture estimation is performed through bundle adjustment using the vertices 6a to 6e of the solid of the marker 6.

[Verification experiment: Flight control by quadcopter]
In the contents of the 6-Dof algorithm, it was explained that the attitude is measured at 5 points, which are the vertices of the quadrangular pyramid, but in the following experiment, it is verified at 16 points including these 5 points. In the verification experiment, a square was selected as the coplanar shape, and a quadrangular pyramid shown in FIG. 2B was selected as the three-dimensional shape.

Two types of markers are placed on the ground and fixed to the ground for attitude estimation. The size of the planar marker was 19 cm × 19 cm × 0.3 cm, and the size of the three-dimensional marker was 19 cm × 19 cm × 12 cm.
For the correct data of 6-DoF attitude for verification, the data using the motion capture device was used.

We used a single marker to move a small unmanned aerial vehicle and recorded the results of 6-DoF attitude estimation. In the case of the planar marker, a large white noise (white noise) was generated with respect to the estimation of _{θ x} , θ _y , t _x , t _y. However, these white noises could be significantly improved in the case of the three-dimensional shape marker.

A comparison of 6-DoF attitude estimation was also performed between the case where two 3D shape markers were used and the case where one stereoscopic marker was used. In this experiment, the positional relationship between the two markers (distance 28 cm) is registered in the attitude control device in advance. It was confirmed that when two three-dimensional markers were used, the white noise was further reduced and the estimation could be performed accurately.

[Advantages / Effects of the present embodiment]
(1) The attitude estimation control using a monocular camera and a three-dimensional marker of a moving body (small unmanned aerial vehicle) that can move freely in three-dimensional space has made it possible for the moving body to fly independently.
(2) Attitude estimation using a monocular camera of a moving object (small unmanned aerial vehicle) that can move freely in three-dimensional space proved that the three-dimensional marker is superior to the planar marker.

[Modification example, etc.]
(1) An embodiment of the present invention has been described by taking a small unmanned aerial vehicle as an example as a moving body. However, the attitude control of the present invention is not limited to this. As a moving body, a small unmanned aerial vehicle freely moves in three-dimensional space (XYZ direction).

The attitude control of the present invention can also be applied to an industrial robot that moves in a two-dimensional space. In this case, the control of the height method (Z direction) from the floor or the like can be accurately measured and controlled by other means, so that the control portion in the Z direction is omitted and the control is two-dimensional (XY direction).

Further, the attitude control of the present invention can also be applied to a truck for moving an article moving in a one-dimensional space, for example, traveling on a straight rail in a factory. In this case, the control portion in the YZ direction is omitted, and the control is one-dimensional (X direction).

(2) By linking the marker with GPS, the relative position representation of the robot and the marker can be converted into the GPS coordinate system.

1: GPS satellite, 2: Small unmanned aerial vehicle, drone, 3: Drone user, 4: Drone maneuvering equipment, 6: Marker, 3D marker, 8: Attitude control device, 10: Board, 14: Triaxial gyro sensor, Memory , 16: CPU, 20: Monocular camera, Single-lens camera,

Claims (6)

In the attitude control system of a moving body that can move in three-dimensional space
A camera and an attitude control device mounted on the moving body,
A solid that has the shape of a cone consisting of the bottom surface of an N-sided polygon (N is an integer value of 3 or more) and N side surfaces, and is convexly installed at a predetermined location so as to face the bottom surface of the pyramid. With markers,
The attitude control device refers to the stereoscopic marker with respect to the region of the bottom surface of the cone of each of the stereoscopic markers in each image data including the region of the stereoscopic marker sequentially imaged by the camera during the flight of the moving object. on the basis of the behavior of the apex region of the cone of the attitude control of the movable body,
Each of the image data is data of an image in a two-dimensional space obtained as a result of being imaged by the camera at each time different in the time direction.
As the attitude control
The position of the apex region of each of the pyramids of the three-dimensional marker at a predetermined time is calculated based on the image data at the predetermined time and the image data at a time before the predetermined time.
The attitude of the moving body is controlled based on the transition of the position of the apex region of the pyramid of the three-dimensional marker at each time of the time, which is obtained as a result of the calculation at each time.
Attitude control system. The attitude control system for a moving body according to claim 1 is
Equipped with a plurality of the three-dimensional markers
The attitude control device relates to a region of the bottom surface of each of the pyramids of the plurality of steric markers in each image data including the regions of the plurality of steric markers. Based on the behavior of the apex region, the posture of the moving body is controlled.
As the attitude control
The position of the apex region of each of the pyramids of the plurality of solid markers at a predetermined time is calculated based on the image data at the predetermined time and the image data at a time before the predetermined time.
Controls the posture of the moving body,
Attitude control system. In the attitude control system for a moving body according to claim 2.
The moving body is an attitude control system, which is an unmanned small airplane. In the attitude control system for a moving body according to any one of claims 1 to 3,
The attitude control device coordinates the bottom surface of one of the three-dimensional markers captured by the camera and the image data of the apex not included in the plane based on the difference between the position known in advance and the position at the time of measurement. Attitude control system that controls three-dimensional attitude by changing to. In the attitude control system for a moving body according to claim 4,
The attitude control device acquires an image of a three-dimensional marker from the camera, converts the acquired image into a scalar image, acquires the contour of a color object from the scalar image, and obtains the contour of the color object from the contour of the color object at the stereoscopic marker and sub-pixel level. A posture control system that detects each vertex and sends it to the marker management unit, performs the initial posture estimation using each vertex of the plane, and performs interactive posture estimation processing through bundle adjustment using each vertex of the solid. In the attitude control system for a moving body according to any one of claims 1 to 5,
The three-dimensional marker is an attitude control system, which is a marker arranged so as to represent a three-dimensional shape with a plurality of luminescent materials.

Images