JP2022092236A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To provide an information processing apparatus that, even when the reliability of a sensor used for position and attitude measurement is reduced due to an impact and vibration, stably achieves the position and attitude measurement.SOLUTION: An information processing method includes: acquiring input information from a sensor mounted on a movable body (S302); acquiring vibration information including the magnitude and direction of vibration applied to the movable body (S303); and based on the input information after correction corrected by using the vibration information, acquiring the position and attitude of the movable body (S304).SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for acquiring the position and posture of a moving body.

Autonomous vehicles such as automated guided vehicles (AGVs) are used in factories and distribution warehouses. In addition, as a method of estimating the position and attitude of such an autonomous vehicle, a method of using a camera or LiDAR (Laser Imaging Detection and Ringing) as a sensor and acquiring the position and attitude difference for each time from the result of measuring the environment around the vehicle. It has been known. Further, a method of calculating a position / attitude value by comparison with environmental map data is known.

Based on the information obtained by the moving object from the sensor, the acquisition accuracy of the relative position between the moving object and the surrounding object moves when creating environmental information indicating the shape and position of the object existing in the environment in which the moving object travels. It may decrease due to body vibration. In Patent Document 1, a technique of determining the reliability of a sensor measured value based on the difference between the measured value of the acceleration applied to a moving body and the value assumed as the acceleration and removing the sensor information at an unreliable time is used. It has been disclosed.

Japanese Unexamined Patent Publication No. 2009-1748998

However, in the method described in Patent Document 1, when the corresponding sensor information is used for the position / attitude measurement, there is a problem that the accuracy of the position / attitude measurement is lowered or the measurement is interrupted at a time when the reliability is low.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing apparatus that can stably obtain the position and posture of a moving body. It also aims to provide the method and program.

The information processing apparatus according to the present invention has the following configuration. That is, the input information acquisition means for acquiring the input information from the sensor mounted on the moving body, the vibration information acquiring means for acquiring the vibration information including the magnitude and direction of the vibration received by the moving body, and the vibration information are used. It has a position / orientation acquisition means for acquiring the position / orientation of the moving body based on the corrected input information corrected by the above.

According to the present invention, it is possible to stably obtain the position and posture of the moving body.

The figure which shows the structure of the mobile body system of 1st Embodiment A block diagram showing a functional configuration of the information processing apparatus of the first embodiment. Flow chart showing the flow of information processing of the first embodiment A flowchart showing the details of the flow of the impact vibration information acquisition process of the first embodiment. Schematic diagram showing the result of feature point tracking in the first embodiment A flowchart showing the details of the flow of the position / orientation calculation process of the first embodiment. The figure which shows the structure of the mobile body system of 2nd Embodiment Flow chart showing the flow of information processing of the second embodiment

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the following embodiments do not limit the scope of claims of the present invention, and all combinations of features described in the following embodiments are essential for constructing the present invention. Is not always.

[First Embodiment]
Hereinafter, the moving body system, the position / posture measuring system, the information processing apparatus, the information processing method, and the computer program according to the first embodiment of the present invention will be described in detail with reference to the drawings.

In the first embodiment, an example in which a moving body equipped with a camera (imaging sensor) as a sensor moves in the environment and the position and orientation of the moving body is estimated will be described. Here, the environment is the area where the sensor has moved and the three-dimensional space around it, and the position / orientation is a value of six degrees of freedom, which is a combination of three-dimensional position information and attitude information of three degrees of freedom. In the following description, the vertical direction of the environment is the z-axis, and the moving body basically travels on the xy plane.

In the present embodiment, it is an object that the camera image is blurred due to the impact or vibration applied to the moving body, and the measurement accuracy of the position / posture by tracking the feature points of the camera image is lowered. The information processing device performs image correction according to the magnitude and direction of impact and vibration, and realizes stable position and orientation measurement.

(Structure of mobile system, position / attitude measurement system and information processing device)
FIG. 1 is a diagram showing an example of the configuration of the mobile system 101 according to the present embodiment. The mobile system 101 includes a position / attitude measurement system 102 including an information processing device 103 and a camera 104, a communication unit 105, a control device 106, and a wheel unit 107.

The information processing apparatus 103 estimates the position and orientation of the moving body based on the input information input from the camera 104. Further, when the moving body is autonomously moving, a movement instruction is given to the control device 106. The details of the configuration of the information processing apparatus 103 will be described later.

The camera 104 is a camera fixed on the mobile system 101 and capable of continuously acquiring a grayscale luminance image. For example, a stereo camera. For the sake of simplicity, it is assumed that the internal parameters such as the focal length and the angle of view of the camera are known, and the image is output in a state where the distortion of the image does not exist or is corrected. Further, the camera 104 shoots at a frame rate of, for example, 30 times per second.

The communication unit 105 receives instructions from the user regarding the movement / rotation of the mobile system 601 and the start / end of the position / orientation measurement process performed by the information processing device 103.

The control device 106 drives the wheel unit 107 based on the instructions of the information processing device 103 and the communication unit 105, and moves and turns the moving body system 101.

The wheel portion 107 is a plurality of tires that are partially or wholly linked to power.

(Configuration of information processing device)
The information processing device 103 has the functions of a general embedded PC device, and is composed of a CPU 111, a ROM 112, a RAM 113, a storage unit 114 such as an HDD or SSD, a general-purpose I / F 115 such as a USB, and a system bus 116.

The CPU 111 uses the RAM 113 as a work memory to execute an operating system (OS) and various computer programs stored in the ROM 112, the storage unit 114, and the like, and controls each unit via the system bus 116. For example, the program executed by the CPU 111 includes a program for executing a process described later.

Further, a sensor for position / attitude measurement (camera 104 in this embodiment), a communication unit 105, and a control device 106 are connected to the information processing device 103 through a general-purpose I / F 115.

(Functional configuration of information processing device)
Hereinafter, the functional configuration of the information processing apparatus according to the present embodiment will be described. The processing of each part shown below is executed as software by reading a computer program from the ROM 112 or the like onto the RAM 113 and then executing the program by the CPU 111. Part or all of the functional configuration may be configured as hardware such as ASIC or FPGA.

FIG. 2 is a block diagram showing a functional configuration of the information processing apparatus 103.

The sensor input information acquisition unit 201 acquires sensor information from the sensor for position / attitude measurement (camera 104 in this embodiment) through the general-purpose I / F 115. In this embodiment, image information is acquired.

The impact vibration information acquisition unit 202 acquires information on the impact and vibration applied to the mobile system 101 based on the sensor information acquired through the general-purpose I / F 115.

The position / orientation acquisition unit 203 is based on the sensor information acquired by the sensor input information acquisition unit 201 and the impact / vibration information acquired by the impact / vibration information acquisition unit 202, and the position / orientation of the moving body system 101 at the time when the sensor information is acquired. Is calculated. The calculated position / attitude information is output to the control device 106 and used for the automatic operation control of the mobile system 101. Alternatively, the position / orientation information can be output to a display device (not shown) and used for displaying the current position of the mobile system on the UI. It is also possible to transmit the sensor information acquired by the sensor input information acquisition unit 201 to the external device via the communication unit 105, and acquire the position / posture calculated by the external device by the position / attitude acquisition unit 205 via the communication unit 105. Is.

(Position / posture measurement processing)
Next, the position / orientation measurement process according to the information processing method using the information processing apparatus of the present embodiment will be described with reference to FIGS. 3 to 6. The processing represented by the flowcharts shown in FIGS. 3 and later is realized by the CPU 111 reading a program stored in the ROM 112 or the storage unit 114 to calculate or process reading information and control each hardware. Note that some or all of the steps shown in each flowchart may be realized by hardware such as ASIC or FPGA.

FIG. 3 is a flowchart showing the flow of the position / posture measurement process in the present embodiment.

In step S301, the information processing apparatus 103 is initialized. The information processing apparatus sets the initial position and orientation with the current position of the mobile system 101 as the origin in the environment and the current posture as a predetermined direction (for example, the positive direction of the Y axis). Further, the sensor input information acquisition unit 201 acquires the first sensor information from each sensor (camera 104 in this embodiment) and holds it in the work memory as "sensor information of the previous time" acquired at the first execution of step S302. do.

Subsequent processes from step S302 to step S304 are repeatedly executed until the user or the like instructs the end of the position / posture measurement. In the following description, a series of processes from one step S302 to step S304 will be described as a process at a certain time t. Here, t is an integer value, and is incremented by 1 each time a series of processes is performed. Further, the description will be made assuming a situation in which the moving body system 101, which has been traveling on a flat surface until time t = 1, has the posture of each sensor greatly swung upward due to an impact at time t = 2. Specific situations of impact may be a situation where a moving object rides on a step such as a sidewalk, a situation where an obstacle such as a stone or a branch is stepped on with a tire, or a situation where an intermittent vibration is received from a stone pavement or the like. ..

In step S302, the sensor input information acquisition unit 201 acquires the latest sensor information from each sensor. In addition, the sensor information of the previous time held in the work memory is acquired.

In step S303, the impact vibration information acquisition unit 202 acquires information on the impact and vibration applied to the moving body system 101. The details of the process of step S303 will be described later.

In step S304, the position / posture acquisition unit 203 calculates the position / posture of the moving body system 101. The details of the process of step S304 will be described later.

Step S305 is a branch of whether or not to end the position / posture measurement process. When the position / posture measurement is terminated by the user's instruction or the like, the process is terminated. If not, the process returns to step S302 and the next time is processed.

(Details of impact vibration information acquisition processing)
Next, the details of the impact vibration information acquisition process shown in S303 will be described. In the present embodiment, the magnitude and direction of the impact and vibration received by the moving body system 101 are analyzed based on the image data acquired by the camera 104. FIG. 4 is a flowchart showing the flow of impact vibration information acquisition processing in the present embodiment.

In step 401, the impact vibration information acquisition unit 202 acquires image data acquired by the camera 104 from each of the latest and previous time sensor information acquired in step S302.

In step S402, the impact vibration information acquisition unit 202 detects and tracks the feature points between the two images acquired in step S401, and acquires the motion vector of the feature points that have been successfully tracked.

FIG. 5A illustrates the result of feature point tracking between the image acquired at time t = 1 and the image at the previous time (t = 0). The penalties in the figure indicate the feature points that were successfully tracked, and the arrows attached to the penalties indicate the amount of movement (motion vector) of the feature points from the previous time.

Further, FIG. 5B illustrates the result of feature point tracking between the image acquired at the next time t = 2 and the image at the previous time (t = 1). .. At time t = 2, the image is blurred in the vertical direction as a result of the impact applied to the camera 104 during shooting. Further, as a result of blurring in the image, some of the feature points that were successfully tracked in FIG. 5A have failed to be tracked.

Further, FIG. 5C illustrates the result of feature point tracking with the image at time (t = 1) after correcting the image blur of the image acquired at time t = 2. ..

In step S403, the impact vibration information acquisition unit 202 compares the motion vector group acquired in step S402 with the motion vector group calculated at the previous time, and determines the magnitude and direction of the impact applied to the moving body system 101. To analyze. Specifically, the difference vector of the motion vector of the feature points that have been successfully tracked at both times is calculated, and the average vector is acquired as the impact vector (impact magnitude and direction) at the corresponding time.

(Details of position / attitude calculation process)
Next, the details of the position / orientation calculation process shown in S304 will be described. In the present embodiment, the position and orientation of the moving body system 101 are calculated based on the image data acquired by the camera 104. The camera 104 is fixed on the moving body system 101, its relative position / orientation is known and held in the RAM 113, and the position / orientation value of the camera 104 and the position / orientation value of the moving body system 101 can be mutually converted. be. For the sake of simplicity, it is assumed that the position and orientation values of the camera 104 and the mobile system 101 are the same in the following description.

FIG. 6 is a flowchart showing the flow of the position / orientation calculation process in the present embodiment.

Step S601 is a branch of whether or not image blur correction is necessary for calculating the position and orientation. Here, if the magnitude of the impact vector acquired in step S403 is equal to or greater than a preset threshold value, it is determined that image correction is necessary, and the process proceeds to step S602. The threshold value of the magnitude of the impact vector is set so that it is judged that the image correction is unnecessary when the moving object goes up a slope or turns a curve. If no image correction is required, the process proceeds to S604. Here, the judgment method is not limited to the size of the impact vector, and for example, it is possible to make a judgment that the image needs to be corrected when the number or ratio of the feature points that have been successfully tracked is less than the threshold value. be.

In step S602, the impact vibration information acquisition unit 202 performs image blur recovery processing on the latest image based on the impact vector acquired in step S403. Specifically, first, the acquired impact vector is decomposed into a vertical component and a horizontal component of the image, and the magnitude of the component in each direction is calculated. Subsequently, the image is transformed into a spatial frequency component using a known two-dimensional Fourier transform. After that, for each spatial frequency, the recovery coefficient is acquired based on a two-dimensional look-up table prepared in advance, which inputs the vertical frequency and the magnitude of the vertical component of the impact vector, and the recovery coefficient is integrated. As a result, the frequency component reduced by the blur caused by the vibration in the vertical direction is recovered. Similarly, the frequency component is recovered in the lateral direction. Finally, a two-dimensional inverse Fourier transform is performed on the spatial frequency component image in which the frequency component has been recovered, and a luminance image in which the frequency component has been recovered is acquired.

In step S603, the position / orientation acquisition unit 203 uses the image corrected in step S602 to detect and track the feature points between the two images again.

In step S604, the position / posture acquisition unit 203 updates the current position / posture of the moving body system 101. Specifically, the position / posture acquisition unit 203 is the position with respect to the previous time of the moving body system 101 based on the feature point detection result in step S603 when the image correction is performed, and in step S402 otherwise. Calculate the posture difference. Specifically, the position / orientation difference is calculated using a known bundle adjustment process. This assumes three-dimensional coordinates in the space for at least eight feature points that have been successfully tracked, and the difference between the reprojection result on both images and the detection position of the feature points on the image. This is a process that optimizes the relative position and orientation between images and the three-dimensional coordinates of feature points so that (reprojection error) becomes small. At that time, if the three-dimensional coordinates of the feature points can be estimated from the stereo distance measurement by the stereo camera and the past feature point tracking results, it can be used as the initial value or the fixed value of the three-dimensional coordinates.

In step S605, the position / posture acquisition unit 203 adds the position / posture difference calculated in step S604 to the position / posture at the previous time, and calculates the position / posture value at the latest time.

By the above processing, even if an impact is applied to the camera used for the position / posture measurement, the position / posture measurement can be stably continued.

[Second Embodiment]
In the first embodiment, an example in which a camera is used alone as a sensor for position / posture measurement and position / posture measurement processing is performed according to impact information acquired from a camera image has been described. In this embodiment, an example of performing position / posture measurement processing by combining a plurality of sensors will be described.

In the present embodiment, as a sensor for position / attitude measurement, position / attitude measurement is performed using the drive amount of the IMU and the tire in addition to the camera. These plurality of measurement methods have different resistance to vibration and impact depending on the principle.

For example, the capacitive accelerometer used in some IMUs has reduced accuracy when a stronger acceleration or continuous vibration than expected is applied. The method of integrating the drive amount of the tire to obtain the movement amount is assumed to move on a flat surface, so the accuracy decreases due to the unevenness of the ground, especially when the tire floats and slips due to a vertical impact such as a step. It becomes difficult to calculate the correct amount of movement. In addition, it is difficult to calculate the correct amount of movement even when the tire slips due to a strong impact in the horizontal direction. At this time, the relationship between the strength of the impact and the degree of decrease in the estimation accuracy of the amount of movement is in the vertical and horizontal directions. Makes a difference. Although the camera causes image blur as described in the first embodiment, it does not have a mechanical structure that is directly affected by shock or vibration, so that it can be said to be a sensor that is relatively resistant to vibration or shock.

In the present embodiment, the contribution of the sensor estimated to have high position / posture measurement accuracy is set relatively high according to the occurrence of vibration or impact, and the result of the position / posture measurement by a plurality of sensors is set according to the contribution. Highly accurate position and orientation measurement is realized by synthesizing.

(Structure of mobile system and position / posture measurement system)
FIG. 7 is a diagram showing an example of the configuration of the mobile system 701 according to the present embodiment. The position / orientation measurement system 702 of this embodiment includes an information processing device 103, a camera 104, and an IMU704. Since the hardware configuration of the information processing apparatus 103 and the camera 104 are the same as those in the first embodiment, the description thereof will be omitted. In the present embodiment, the camera 104, the control device 106, and the IMU704 function as sensors for position and orientation measurement.

The IMU704 is an inertial sensor fixed on the mobile system 701. Like the camera 104, the IMU 704 is connected to the information processing apparatus 103 through the general-purpose I / F 115. The IMU704 acquires the three-dimensional acceleration and angular velocity applied to the mobile system 701. On the other hand, when a strong acceleration exceeding the threshold value is applied, the acquisition accuracy of the acceleration and the angular velocity is lowered.

For the sake of simplicity of explanation, it is assumed that the IMU 704 is attached to the center of the moving body system 701. In the following description, the position / attitude / acceleration / angular velocity of the moving body system 701 and the position / orientation / acceleration / angular velocity of the IMU 704 are the same. Suppose there is.

Further, both the camera 104 and the IMU 704 are fixed on the mobile system 701, their relative positions and orientations are known, and they are held in the RAM 113. Therefore, the position / orientation information of the camera 104 and the position / attitude information of the IMU 704 (that is, the position / attitude information of the moving body system 701) can be converted into each other. For the sake of simplicity, in the following description, it is assumed that the position / orientation information measured by using the camera 104 indicates a value converted into the position / attitude information of the IMU704.

Since the functional configuration of the information processing apparatus 103 according to the present embodiment is the same as the configuration of the first embodiment shown in FIG. 2, detailed description thereof will be omitted. Further, since the basic flow of the position / posture measurement process and the details of the impact vibration information acquisition process are the same as those of the first embodiment shown in FIGS. 3 and 4, the detailed description thereof will be omitted.

(Details of position / attitude calculation process)
Hereinafter, the details of the position / orientation calculation process shown in S304 in the present embodiment will be described. In the present embodiment, the position and orientation of the mobile system 701 are calculated based on the data acquired from the camera 104, the control device 106, and the IMU704.

FIG. 8 is a flowchart showing the flow of the position / orientation calculation process in the present embodiment. Further, the processes of steps S801 to S803 are processes that are synchronously executed in parallel for each sensor.

In step S801, the position / posture acquisition unit 203 calculates the position / posture difference value from the previous time based on the camera 104, as in the first embodiment. Since this process is the same as the process of steps S601 to S604 described in the first embodiment, detailed description thereof will be omitted.

In step S802, the position / orientation acquisition unit 203 integrates the acceleration and angular velocity values obtained between the acquisition of the previous image by the camera 104 and the acquisition of the latest image, and the position based on the IMU. Calculate the posture difference value.

In step S803, the position / posture acquisition unit 203 integrates the tire drive amount obtained between the time when the camera 104 acquires the previous image and the time when the camera 104 acquires the latest image, and adds the tire drive amount to the tire drive amount. Calculate the position-posture difference value based on this. This position-posture difference value is calculated on the assumption that the moving body system 101 is traveling on a plane. Therefore, this position-posture difference value is composed of a movement vector on an xy plane corresponding to a plane such as a floor and a rotation amount with respect to the z-axis perpendicular to the plane.

In step S804, the position / attitude acquisition unit 203 determines the contribution of each sensor used when synthesizing the position / attitude difference value calculated based on each sensor. The calculation of the contribution is determined based on the impact vector calculated by the impact vibration information acquisition unit 203. Specifically, the acquired impact vector is decomposed into a vertical component and a horizontal component of the image, and the contribution value is based on the size of the component in each direction and the calculation method set in advance for each sensor. To decide.

The method of calculating the contribution of each sensor uses a predetermined function with the magnitude of impact or vibration of each axis as an argument. In the present embodiment, since the camera 104 is considered to be more resistant to vibration and shock than other sensors, the sensor is used as a reference for position / posture measurement, and the contribution of the camera 104 is fixed to 1.

The accuracy of the IMU 704 is reduced when an acceleration of a predetermined strength or more is applied regardless of the direction of vibration or impact acquired from the image of the camera 104. Therefore, the contribution of the IMU704 refers only to the magnitude of the impact vector, and the magnitude of the impact vector decreases linearly between the thresholds α1 and α2, the contribution is 1 when the impact vector is less than or equal to the threshold value α1, the contribution is 0 when the threshold value is α2 or more. Use a function like. The threshold values α1 and α2 are values defined according to the vibration resistance performance of the IMU used.

The position-posture difference measurement based on the driving amount of the tire takes a method of calculating the contribution degree for each of the vertical component and the horizontal component of the impact vector and integrating them. In the vertical direction, the accuracy is reduced due to the unevenness of the ground, and especially when the tire slips due to an impact, the accuracy is greatly reduced. Therefore, when the vertical component of the impact vector is 0, the contribution degree is 1, and when the threshold value β1 or more, the contribution degree is 0, and a function that decreases linearly between 0 and the threshold value β is used. The threshold value β is a value defined as the magnitude of the vertical component of the impact vector when an impact equivalent to a gravitational acceleration of 1 G is applied to the moving body system 101 in the vertical direction.

In the lateral direction, it is considered that the impact in the range where the tire does not slip does not have a great adverse effect on the position / attitude measurement based on the driving amount, but when slip occurs, it becomes impossible to calculate the accurate position / attitude difference. .. Therefore, when the lateral component of the impact vector is less than the threshold value γ, the contribution degree is set to 1, and when it is equal to or more than the threshold value, it is set to 0. The threshold value γ is a value defined based on the coefficient of friction between the tire and the assumed ground.

In step S805, the position / posture acquisition unit 203 sets the position / posture difference value of the moving body system 701 based on the position / posture difference value calculated in steps S801 to S803 based on each sensor and the contribution of each sensor calculated in step S804. Is calculated. Basically, the position / orientation difference calculated by each sensor is divided into the position difference on the x, y, and z axes and the amount of rotation with respect to the x, y, and z axes, and the contribution of each sensor is used as the weight value for each. Calculate the weighted average value. Here, since the position / posture difference value based on the drive amount of the tire does not have information on the position difference in the z-axis direction and the rotation amount with respect to the x-axis and the y-axis, their weights are treated as 0.

In step S806, the position / posture acquisition unit 203 adds the position / posture difference calculated in step S805 to the position / posture at the previous time, calculates the position / posture value at the latest time, and determines the position / posture.

By the above processing, even when the position / orientation measurement processing is performed by combining a plurality of sensors, it is possible to suppress the decrease in accuracy due to impact or vibration and perform the position / attitude measurement in a stable manner.

[Modification example]
In the first embodiment, as a position / posture measurement method using a camera, a method of adding the position / posture difference calculated from the feature point tracking results of a plurality of images to the initial position / posture value has been described as an example. The position / attitude measurement method used is not limited to this. For example, a known position / orientation measuring method such as SLAM (Simultaneus Localization and Mapping) or SVO (Semi-Direct Monocular Visual Odometry) may be used.

The case of using SLAM will be specifically described. The moving body system holds information on a three-dimensional point cloud indicating the shape characteristics of an object in the environment in which the moving body on which the camera is mounted travels as map information of the environment. The information of the three-dimensional point cloud is the three-dimensional coordinates in the world coordinates of the image feature points. The map information of the environment is generated in advance based on the image obtained by capturing the shape of the object existing in the environment, and is stored in the storage device. Based on the input image and the map information, the position / orientation acquisition unit 203 uses the positions of the feature points (edge features indicating the shape) on the input image and the position / orientation of the camera when the input image is captured on the map. Minimize the difference in position (reprojection error) when the feature points of are projected onto the input image. This estimates the position and orientation of the camera.

Further, an environmental map creation system that creates an environmental map around a moving body at the same time as measuring the position and orientation by using these methods is also within the scope of the present invention.

In the second embodiment, a method of measuring the position and posture based on the sensors of the camera, the IMU, and the tire drive amount and synthesizing the results based on the vibration impact information has been described, but the sensors that can be used are limited to these. Not done. For example, LiDAR (Laser Imaging Detection and Ranging) may be added to the sensor, or a part thereof may be replaced. Generally, since the mirror rotation type LiDAR has a mechanical rotation mechanism, the accuracy tends to decrease due to an impact, and it is better to set a lower contribution than the camera when vibration or an impact occurs. On the other hand, the MEMS (Micro Electro Electrical Systems) type LiDAR does not have a mechanical mechanism. Since the image blur is small like a camera, a higher contribution than the camera is set for a strong impact.

Alternatively, the position information acquired by GNSS (Global Navigation Satellite System) can be used, or the position / attitude value calculated from the result of observing the moving body system with a sensor such as a camera installed outside the moving body can be used. good. For example, a method of processing the image of a surveillance camera that captures a moving moving body system on a server machine, calculating the position and orientation information of the moving body system recognized as an image in the image, and transmitting it to the moving body system through a communication unit. Can be considered.

In the second embodiment, the contribution of each sensor was calculated based on a predetermined mathematical formula. However, the method of calculating the contribution is not limited to this, and any method may be used as long as it is a method of setting a relatively high contribution to a sensor having a high position / attitude measurement accuracy estimated from the acquired vibration and impact information. For example, a two-dimensional look-up table may be used in which the magnitudes of the vertical component and the horizontal component of the impact vector are input. Alternatively, the contribution may be determined based on the history of the impact vector at each of the latest times as well as the impact vector acquired at the corresponding time.

Alternatively, the contribution may be determined by adding information other than vibration and impact information. For example, it may be considered that the accuracy of the position / posture measurement based on the tire drive amount and the GNSS is relatively lower than that of other sensors even in the ideal state, and a process of setting the contribution to a lower degree may be performed. Alternatively, in a vehicle having front wheels and rear wheels, a certain amount of slippage occurs in the tires during steering operation, so the contribution of position / attitude measurement based on the driving amount of the tires is performed while controlling the curve of the moving body. You may perform the process of lowering.

In the first embodiment and the second embodiment, the feature point tracking result of the camera image is used as the impact vibration acquisition means, but the impact vibration acquisition means that can be used in the present invention is not limited to this. Further, in the first embodiment and the second embodiment, the direction is acquired in two axes in addition to the magnitude of the impact and the vibration, but the acquisition of the direction of the impact and the vibration is not indispensable, and the form in which the direction is acquired. Is not limited to this. For example, the acceleration value may be acquired from the IMU on three axes. Alternatively, only the magnitude of impact or vibration may be acquired.

Further, the acquisition of shock and vibration information may be performed by using information other than the information of the sensor on the mobile system. For example, the image of an external device such as a surveillance camera that captures the moving mobile system from an observable position is processed on the server machine, and the swing condition of the mobile system recognized as an image in the image is analyzed. It may be transmitted to the mobile system through the communication unit. The vibration condition is obtained by tracking the feature points obtained by analyzing the image. As the feature point tracking method, the method described in step S402 can be used. Alternatively, the magnitude of vibration or impact predicted in association with the position / orientation information may be acquired from the electronic map data stored in the information processing device. Specifically, the position information of the step or the rough road recorded on the map is acquired, and the position / posture measurement is performed assuming that an impact or vibration of a predetermined magnitude is generated at that location.

When blur correction processing is performed on an image as in the first embodiment, when information on 3-axis impact or vibration is acquired, information on 3-axis is projected on an image plane and 2-axis impact or vibration is performed. By converting to the information of the above, the blur correction process can be carried out in the same manner as in the first embodiment. Alternatively, when information on only the magnitude of impact or vibration is acquired, blur correction processing is performed without distinguishing the direction on the image.

When setting the contribution of each sensor according to the vibration or impact information as in the second embodiment, a function or look-up table according to the number of axes of the acquired vibration or impact information is used. The contribution may be calculated. Alternatively, the contribution may be calculated based on the vibration or impact information converted to a smaller number of axes.

In the first embodiment and the second embodiment, the moving body system including a plurality of tires as the wheel parts has been described as an example, but the wheel parts of the moving body system are not limited thereto. For example, it may be a flying object such as a multicopter, or it may be a walking object having a plurality of legs. Further, it may be a manned mobile body that the user can board and operate directly. Further, it may be a moving body having a function of autonomously traveling on a preset route.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Mobile system 102 Position / posture measurement system 103 Information processing device 104 Camera 105 Communication unit 106 Control device 107 Wheel unit 201 Sensor input information acquisition unit 202 Impact vibration information acquisition unit 203 Position / posture acquisition unit

Claims (10)

Input information acquisition means for acquiring input information from sensors mounted on moving objects, and
A vibration information acquisition means for acquiring vibration information including the magnitude and direction of the vibration received by the sensor due to the vibration of the moving body, and a vibration information acquisition means.
An information processing apparatus comprising: a position / posture acquisition means for acquiring the position / posture of the moving body based on the corrected input information corrected by using the vibration information. When the moving body is subjected to vibration of a magnitude exceeding the threshold value, the position / orientation acquisition means corrects the input information acquired by the input information acquisition means by using the vibration information, and then inputs the corrected input. The information processing apparatus according to claim 1, wherein the position and orientation of the moving body are acquired based on the information. The vibration information acquisition means is characterized in that it acquires the vibration information by tracking the feature points obtained from the image information of the moving body captured by an external device capable of observing the moving body. Or the information processing apparatus according to 2. The input information acquisition means acquires image information from an image pickup sensor mounted on the moving body, and obtains image information.
The information processing apparatus according to claim 1 or 2, wherein the vibration information acquisition means acquires the vibration information by tracking feature points obtained from the image information. When the ratio of the feature points that can be traced is equal to or less than the threshold value, the position / posture acquisition means of the moving body receives the input information based on the corrected input information corrected by using the vibration information. The information processing apparatus according to claim 4, wherein the position and orientation are acquired. Further having a means for acquiring map information of the environment in which the moving body moves,
The information processing apparatus according to any one of claims 1 to 5, wherein the vibration information acquisition means acquires the vibration information based on the environmental information included in the map information. Input information acquisition means for acquiring input information from each of multiple sensors mounted on the moving object,
Vibration information acquisition means for acquiring vibration information including the magnitude and direction of vibration received by the moving body, and
Information characterized by having a position / posture determining means for determining the position / posture of the moving body by weighting the position / posture of the moving body acquired by using each of the input information based on the vibration information. Processing equipment. The input information acquisition process to acquire the input information from the sensor mounted on the moving body, and
A vibration information acquisition process for acquiring vibration information including the magnitude and direction of vibration received by the sensor due to the vibration of the moving body, and a vibration information acquisition process.
An information processing method comprising: a position / posture acquisition step of acquiring the position / posture of the moving body based on the corrected input information corrected by using the vibration information. Input information acquisition process to acquire input information from each of multiple sensors mounted on the moving body,
A vibration information acquisition process for acquiring vibration information including the magnitude and direction of vibration received by the sensor due to the vibration of the moving body, and a vibration information acquisition process.
Information characterized by having a position / posture determination step of weighting the position / posture of the moving body acquired by using each of the input information based on the vibration information to determine the position / posture of the moving body. Processing method. A program that causes a computer to function as each means of the information processing apparatus according to any one of claims 1 to 7.

Images