JP6441586B2 - Information processing apparatus and information processing method - Google Patents

Description

  Some embodiments according to the present invention relate to an information processing apparatus and an information processing method.

  In recent years, for example, it has been considered that various processing such as position estimation is performed in real time by processing video captured by a camera (image sensor) or the like in real time. In particular, a flying robot such as a multicopter robot among UAVs (Unmanned Aero Vehicles), unlike a ground robot, cannot use contact with external fixed objects such as wheel odometry information, and thus performs self-position estimation. For this purpose, it is necessary to use a non-contact method. Therefore, self-position estimation by image processing is often used for flying robots.

  For self-position estimation, for example, it may be necessary to detect a feature point from an image taken by a camera or generate a descriptor describing the feature point. As a technique for detecting feature points, for example, there is a technique called FAST (see Non-Patent Document 1), and as a technique for generating a descriptor describing a feature point, for example, BREF (see Non-Patent Document 2). There is a technique called.

  When self-position estimation is performed on a flying robot by image processing, a computer on the aircraft is usually insufficient in resources, so a ground station computer is often required. This is because calculation based on online image processing and periodic attitude estimation and control are difficult to perform in real time with a computer on the aircraft, so such calculations are performed after wirelessly communicating with a ground station computer. This is a method of performing a large amount of calculations on the ground station side.

E. Rosten, R. Porter, and T. Drummond. "Faster and better: A machine learning approach to corner detection." IEEE Trans. Pattern Analysis and Machine Intelligence, 32: 105-119, 2010. M. Calonder, V. Lepetit, C. Strecha, and P. Fua. "Brief: Binary robust independent elementary features." In European Conference on Computer Vision, 2010.

  However, when a ground plane is used, the active range of the flying robot is limited to a range where infrastructure for wireless communication is in place. In addition, if the number of flying robots increases, the number of ground planes must increase proportionally. If a flying robot system specialized in so-called on-board processing that performs various processes such as attitude control on the aircraft is possible, the communication distance can be determined by the robot judging and acting on its own without relying on ground planes. It is possible to perform missions in a wide range of environments that are not bound by the above.

  Some aspects of the present invention have been made in view of the above-described problems, and an object of the present invention is to provide an information processing apparatus and an information processing method capable of performing position estimation by real-time processing with limited resources. I will.

  An information processing apparatus according to a predetermined aspect of the present invention includes an input process for receiving image frame data input at a constant rate, a detection process for detecting a plurality of feature points by parallel processing for the input image frame, and A first calculation processing module capable of executing description processing for generating a plurality of descriptors respectively describing features of the plurality of feature points by parallel processing at a constant speed by control in units of clocks; and the first calculation A process for obtaining correspondence between image frames for the plurality of feature points of each image frame detected and described by a processing module, and a position estimation process for performing position estimation based on the correspondence 2 calculation processing modules.

  An information processing method according to a predetermined aspect of the present invention includes an input process of receiving image frame data input at a constant rate, a detection process of detecting a plurality of feature points by parallel processing for the input image frame, and A description process for generating a plurality of descriptors respectively describing features of the plurality of feature points by parallel processing is executed by a first calculation processing module that can be executed at a constant speed by control in units of clocks. For a feature point of each described image frame, a second calculation processing module capable of executing a process for obtaining a correspondence between image frames and a position estimation process for performing position estimation based on the correspondence are executed. .

  In the present invention, “part”, “means”, “apparatus”, and “system” do not simply mean physical means, but “part”, “means”, “apparatus”, “system”. This includes the case where the functions possessed by "are realized by software. Further, even if the functions of one “unit”, “means”, “apparatus”, and “system” are realized by two or more physical means or devices, two or more “parts” or “means”, The functions of “device” and “system” may be realized by a single physical means or device.

It is a block diagram which shows the structural example of the information processing apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the specific example of a feature point detection process. It is a figure for demonstrating the specific example of a feature point detection process and a descriptor production | generation process.

  Embodiments of the present invention will be described below. In the following description and the description of the drawings to be referred to, the same or similar components are denoted by the same or similar reference numerals.

  In the present embodiment, an information processing apparatus that is a palm-sized flying robot that realizes self-position estimation and fully automatic flight operation using a monocular camera by on-board processing that does not require a ground plane will be described. Note that the information processing apparatus implementation method is not limited to the palm-sized flying robot according to the present embodiment, and can be implemented in various other forms without departing from the gist of the present invention. The present embodiment is merely an example in all respects, and the specific configuration and size are not limitedly interpreted.

(1 Outline of functional configuration)
The information processing apparatus according to the present embodiment can be largely devised in order to realize such on-board processing. The first is to realize a small electronic circuit having powerful information processing performance by using a combination of a Field-Programmable Gate Array (FPGA) and a Micro-Control Unit (MCU). Secondly, in order to realize the two required specifications of the control of the aircraft and the execution of the mission on the information processing device which is a palm-sized flying robot with limited resources, the signal processing part is parallelized and By cascading, the software is designed from the bus level, minimizing the cost of memory transfer between modules.

  FIG. 1 is a functional block diagram showing an outline of a functional configuration of an information processing apparatus 1 that is a flying robot according to the present embodiment. In the information processing apparatus 1 in FIG. 1, an on-board self-position estimation function and a tracking function to a target position are realized. As illustrated in FIG. 1, the information processing apparatus 1 according to the present embodiment includes an image sensor 10, an FPGA 20, an MCU 30, an inertial measurement device (hereinafter also referred to as an IMU) 40, and a motor 50.

  The image sensor 10 is a sensor that functions as a monocular camera, and outputs image frames captured at a constant rate. Various specs that can realize the image sensor 10 can be considered. Here, the image frame has a color YUV422 format with a size of 160 pixels × 120 pixels, an image frame update rate of 30 fps (frames per second), and an operating frequency of 25 MHz. Can be used. For example, a CMOS camera can be used.

  The IMU 40 can include, for example, a three-axis acceleration sensor, a three-axis gyro sensor, and a three-axis compass sensor (magnetic sensor), and using the values detected by these sensors, It is possible to calculate the pitch and yaw angle (sensor fusion). Various sensors may be used for various sensors included in the IMU 40. For example, a combination of Kionix KXRB5-2050 for a three-axis acceleration sensor and a combination of LPR4150AL and LY3100ALH for a three-axis gyro sensor by STMicroelectronics HMC5883L manufactured by Honeywell can be used for the three-axis compass sensor.

  The motor 50 is for rotating a propeller (not shown) for flying the information processing apparatus 1 which is a flying robot. For example, when the information processing apparatus 1 is a quadcopter having four propellers, the information processing apparatus 1 has four motors. There are roughly two types of motors, brush type and brushless type, and either can be used for the motor 50, but here the KV value (rotation speed per voltage) is not used without using a gear reduction mechanism. Use a relatively small brushless motor. Various brushless motors that can be used as the motor 50 are conceivable. For example, it is conceivable that a brushless motor HXM2000 manufactured by hexTronik is used as the motor 50 and a 4-inch propeller (not shown) is directly driven.

  Among the components assembled in the information processing apparatus 1, the FPGA 20 and the MCU 30 are calculation modules that actively perform calculation processing. Note that the FPGA 20 and the MCU 30 can be realized as physically different processing chips (processors), or can be realized physically on one processing chip.

  Here, when performing processing on an image frame input from the image sensor 10, a feature point detection process for detecting a plurality of feature points for the image frame after temporarily storing the image frame in a memory such as a DDR SDRAM. It is also conceivable that the MCU 30 performs most of the necessary calculation processing including the descriptor generation processing 25 (each described later) for generating the descriptors 23 and descriptors of each feature point. However, when trying to store the image frame in the memory, a delay occurs due to processing for waiting for synchronization of the camera signal. In addition, attempting to implement most of the processing on the MCU 30 that is a single processor also leads to a delay in the computation time. Such processing time delay is fatal in realizing a flying robot that needs to operate in real time.

  Therefore, the information processing apparatus 1 according to the present embodiment does not perform image frame storage processing in the DDR SDRAM, and performs feature point detection processing 23 and descriptor generation processing 25 on the FPGA 20.

(2 Processing in FPGA 20 and MCU 30)
(2.1 Overview)
Hereinafter, an outline of processing in the FPGA 20 and the MCU 30 will be described. The image frames from the image sensor 10 are stream-input at a constant rate (here, 30 fps) by the input processing 21 of the FPGA 20. The FPGA 20 performs in parallel a feature point detection process 23 for detecting a plurality of feature points and a descriptor generation process 25 for generating a descriptor of each feature point for each input image frame. In this embodiment, the FAST algorithm is used for the feature point detection process 23, and the BRIEF algorithm is used for generating a feature point descriptor (hereinafter also referred to as a BRIEF descriptor). In the present embodiment, the BRIEF descriptor is 256 bytes.

  Next, the FPGA 20 outputs the feature point information including the coordinates of the detected feature point on the two-dimensional image coordinates and the descriptor of the feature point to the SRAM built in the MCU 30 by the output process 27 (DMA (Direct Memory Access)). Forward. At this time, 8 Kbytes are secured as a ring buffer on the SRAM, and the output process 27 is immediately performed by the FPGA 20 as soon as it is determined by the feature point detection process 23 that a feature point has been found. The feature point information output from the FPGA 20 by the output process 27 includes a frame number, and the MCU 30 can detect the frame number of the feature point. When the process for one frame of the feature point detection process 23 and the descriptor generation process 25 is completed, the output process 27 notifies the MCU 30 that all the feature point processes have been completed.

  There are roughly two processes of the MCU 30. The first is a group of functions that are called periodically by a timer interrupt. Here, such processing is called a control process. In the control process, the MCU 30 acquires values from the IMU 40 including an acceleration sensor, a gyro sensor, and a compass sensor, and obtains an attitude angle by sensor fusion, and a target angle calculated from the current position and the target position. This includes attitude control processing 33 for performing PID (Proportional Integral Derivative Controller) control.

  The second is a group of functions that are executed in the idle time after the timer interrupt, and the feature point management process 37 that records the feature points transferred from the FPGA 20 and their descriptors, and the past feature points. A feature point associating process 35 that takes a correspondence relationship, and a bundle adjustment process 39 that performs self-position estimation based on the correspondence relationship and values from the MCU 30 are included.

(2.2 Feature Point Detection Process 23 and Descriptor Generation Process 25)
Hereinafter, the feature point detection process 23 and the descriptor generation process 25 performed by the FPGA 20 in the present embodiment will be described. The feature point detection processing 23 and the descriptor generation processing 25 in this embodiment are collectively referred to herein as “CSFB (Concurrently Streamed FAST and BREF). The features of CSFB include the following four points.
1. The camera's depth direction and rotational movement can also be tracked.
2. It can be executed in real time (for example, 30 fps or more).
3. No external memory is required to store raw image frame data.
4). Since no multiplication or division processing is performed, no DSP (Digital Signal Processor) resource is required on the FPGA 20.

  Hereinafter, a process executed on the FPGA 20 will be described. In the FPGA 20, each process operates independently of each other according to a given clock, and signal efficiency can be greatly improved by parallelizing the processes.

  As described above, the image sensor 10 according to the present embodiment has a YUV422 format, 160 × 120 pixels, can obtain an image frame at a frame rate of 30 fps, and is assigned 16 bits per pixel and 2 clocks. On the FPGA 20, the FAST algorithm (feature point detection processing 23) and the BREF algorithm (descriptor generation processing 25) are executed simultaneously.

  First, the FAST algorithm (feature point detection processing 23) will be described with reference to FIG. The feature point detection method using the FAST algorithm is represented by 16 pixels (Fast Scan Point, “F” in FIG. 2) around the target pixel (Target Point, expressed by “C” in FIG. 2). ) Is compared with the target pixel at the center, and a predetermined number of pixels (here, 10 pixels) in which the number of pixels brighter than the threshold / dark pixels continuous in the circumferential direction is determined in advance. When the above is reached, the target pixel is detected as a feature point.

The brightness information of the new pixel is sent at a rate of 8 bits once every two clocks (Newdata [7: 0] in FIG. 2), and the buffer Ibuf (l) [j] (0) having a size of 8 × 8 × 8 bits. ≦ l ≦ 7, 0 ≦ j ≦ 7). In addition, eight block RAMs (BRAM (l) [m], 0 ≦ l ≦ 7, 3 ≦ m ≦ hpix-3) perform writing of new pixel data and reading of past data. Is used to supply Ibuf with pixel information around the target point. The eight block RAMs (BRAMs) change the role of data writing or data reading in accordance with the rising edge count of the horizontal synchronization signal (HSYNC).
The above process can be expressed as an expression as follows.

ここで、「！ｌ」はｌではないことを意味し、「％８」は８で割った剰余を意味する。

Here, “! L” means not l, and “% 8” means the remainder divided by 8.

  Such processing for determining whether or not the target pixel is a feature point is performed on each Fast scan point around the target pixel, as shown in FIG. When the Fast scan point is 16 pixels, there are a total of 32 evaluation patterns (a pattern in which 10 pixels that are brighter than the target pixel are consecutive, starting from each of the 16 pixels, and darker than the target pixel. A pattern of ten consecutive pixels, which is fbp [i] and fbn [i] (0 ≦ i ≦ 15) in FIG.

Such feature point determination processing is speeded up by machine learning using a tree structure in the original FAST algorithm. However, in the present embodiment, the FPGA 20 evaluates all patterns by parallel processing for such evaluation (calculation for evaluating whether or not it is a feature point). When it is determined that each of the patterns is a feature, a positive value is given as an evaluation value, and the total value of the previous pattern becomes an evaluation value fscore [cy] [cx] as a feature point of the target pixel. .
The above process can be expressed as an expression as follows.

  As described above, the descriptor generation process 25 is performed on the detected feature point by the BREF algorithm. In the BRIEF algorithm, a point that randomly extracts two points around a target pixel of interest, repeats the process of comparing the brightness and darkness, and describes a feature point using a binary string indicating the result (BRIEF descriptor. FIG. 3 is desc [i]). Here, the extraction process is performed 256 times for one target pixel, thereby generating a 64-byte descriptor indicating the result. The processing can be executed in parallel by the FPGA 20.

In this embodiment, the FPGA 20 performs the feature point descriptor generation process 25 at the same time as the feature point detection process 23, so that it is possible to obtain a BREF descriptor at the same time as the feature point is detected. As a result, the FPGA 20 can perform feature point detection and description processing from an image frame as stream data without using an external memory.
The above process can be expressed as an expression as follows.

Here, the combination of k _{0 to} k ₃ is selected from 256 combinations randomly selected on the desktop computer, and the combination with the best performance compared with the correct answer data is selected. Hardware coating can be performed as an initial value.

  In the FPGA 20, when the BRIEF descriptor (BRIEF binary string) obtained as 256 bits is determined as a feature point by the feature point detection processing 23 (FAST algorithm), that is, when fscore [cy] [cx] is not 0. As an output process 27 (data transfer process), data transfer to the MCU 30 is performed. The feature point information sent to the output process 27 can include a 2-byte feature point coordinate, a 32-byte BREF descriptor, a 1-byte index, and fscore.

  The feature point information is transferred to the MCU 30 by the output processing 27 using the DMA transfer function of the MCU 30 as described above. Thereby, the MCU 30 can obtain the coordinates of the feature points and the information of the descriptor without using any CPU resources.

  In the above method, the data transfer of the FPGA 20 is executed regardless of the data write command and the data read command of the MCU 30, so that the waiting time for synchronizing with the vertical synchronization signal can be set to zero. is there.

(2.3 Processing in MCU 30)
The MCU 30 receives the feature point information from the FPGA 20 as described above. Among these are the coordinates of feature points in the image plane and the BRIEF descriptor. Also, in the control process that is periodically called by the timer interrupt, the MCU 30 obtains the rotation posture by the sensor fusion process 31. Further, the MCU manages the feature points obtained from the FPGA 20 (feature point management processing 37), describes the correspondence (feature point association processing 35), and performs self-position estimation (bundle adjustment processing 39). .

  Examples of the sensor fusion process 31 include a complementary filter, an extended Kalman filter, a filter using a quaternion-based gradient method, and any of these may be used. In this embodiment, it is possible to use the quaternion-based gradient method proposed by Madgwick et al.

  The feature point management process 37 will be described. Some feature points always appear in the image frame, while others disappear (cannot be detected). In particular, many feature points obtained from an image frame photographed from a camera (image sensor 10) mounted on a flying robot often repeat blinking due to vibration caused by flight. Therefore, if a feature point that has temporarily disappeared appears again in a later image frame, it must be recognized.

  However, if feature points that have disappeared are stored indefinitely, memory features increase monotonically as new feature points appear as they are added, and thus cannot be implemented with limited memory resources. Therefore, in the present embodiment, the correspondence between the feature points managed so far and the feature points obtained from the latest image frame is obtained (feature point association processing 35), and the association is performed. Penalties are given to those that fail, and feature points that reach a certain score are deleted from the memory. In addition, when the data amount of the stored feature points reaches the upper limit of the memory, data is prevented from overflowing by not adding new feature points.

  Initialization of each feature point is performed using a plurality of frames in which the same feature point is captured. There are methods called the 8-point method and the 5-point method for obtaining the rotation and translation of the camera using two frames. However, in order to perform such a calculation on the MCU 30, the calculation cost becomes very high. Use a different approach. In the present embodiment, by using the assumption that the ground shape is a plane and performing bundle adjustment processing 39 on the feature point coordinates, the feature points are gradually brought closer to the actual three-dimensional coordinates of the feature points. Initialize the.

  When it is assumed that the ground shape is a plane, the feature point coordinates can be obtained based on the known current position. By performing bundle adjustment over several frames using this as an initial value, it is possible to calculate the three-dimensional coordinates of actual feature points.

  Finally, the self-position initialization and estimation method will be described. As a starting point for self-position estimation, a state is assumed in which a down-facing camera is used to fly a certain height above a planar ground surface. About this initial height, the scale of an image coordinate system and the scale of an actual coordinate can be match | combined previously using an ultrasonic sensor etc. Using this, the feature point coordinates reflected on the plane are determined. Subsequent frames are updated from the bundle adjustment formula for the self-position using the feature points that have been initialized. Note that new feature points do not contribute to the estimation of self-location.

(3 Effects of this embodiment)
The information processing apparatus 1 that is the flying robot according to the present embodiment described above can fly by performing self-position estimation in real time on board (limited resources). In particular, the information processing apparatus 1 performs feature point detection and description processing using the FPGA 20. The FPGA 20 can execute processing at a constant speed according to the input of an image frame by controlling in units of clocks, can execute processing related to feature point detection and identifier generation in parallel, has low power consumption, etc. Therefore, it can significantly contribute to self-position estimation in real time.

1: Information processing apparatus 10: Image sensor 20: FPGA
21: Input processing 23: Feature point detection processing 25: Descriptor generation processing 27: Output processing 31: Sensor fusion processing 33: Attitude control processing 35: Feature point association processing 37: Feature point management processing 39: Bundle adjustment processing 50: motor

Claims (7)

Input processing that receives image frame data input at a constant rate,
A detection process for detecting a plurality of feature points by performing parallel processing of an operation for evaluating whether or not the input image frame is a feature point, and a feature of each of the plurality of feature points are described. A first calculation processing module capable of executing description processing for generating a plurality of descriptors by parallel processing at a constant speed by control in units of clocks;
A process for obtaining a correspondence between image frames for the plurality of feature points of each image frame detected and described by the first calculation processing module, and a position estimation process for estimating a position based on the correspondence An information processing apparatus comprising: a second calculation processing module capable of executing The first calculation processing module executes the detection processing and the description processing simultaneously.
The information processing apparatus according to claim 1. In the detection process, the first calculation processing module detects the plurality of feature points by a FAST algorithm.
The information processing apparatus according to claim 1 or 2. The first calculation processing module generates the plurality of descriptors describing features of the plurality of feature points by a BREF (Binary Robust Independent Elementary Features) algorithm in the description processing.
The information processing apparatus according to any one of claims 1 to 3. The second calculation processing module performs position estimation based on the input from the inertial measurement device and the correspondence relationship.
The information processing apparatus according to any one of claims 1 to 4. The first calculation processing module and the second calculation processing module are mounted on the same chip.
The information processing apparatus according to any one of claims 1 to 5. Input processing that receives image frame data input at a constant rate,
A detection process for detecting a plurality of feature points by performing parallel processing of an operation for evaluating whether or not the input image frame is a feature point, and a feature of each of the plurality of feature points are described. A description process for generating a plurality of descriptors by parallel processing is executed by the first calculation processing module that can be executed at a constant speed by control in clock units, and
A process for obtaining a correspondence between image frames for the plurality of feature points of each image frame detected and described by the first calculation processing module, and a position estimation process for estimating a position based on the correspondence An information processing method for causing a second calculation processing module capable of executing

Images