JP7064948B2 - Autonomous mobile devices and autonomous mobile systems - Google Patents

Description

The present invention relates to techniques for autonomous mobile devices and autonomous mobile systems.

Conventionally, technologies related to autonomous movement have been developed in which external sensors and internal sensors are mounted on moving objects such as robots and automobiles to detect and avoid obstacles around the moving objects while autonomously moving to the destination. ing. Here, the external sensor is an image sensor, LiDAR, or the like, and the internal sensor is an encoder, an inertial measurement unit, or the like. Under these circumstances, there is an increasing demand for technological development using relatively inexpensive image sensors in order to reduce the cost of mounted sensors.

For example, Patent Document 1 states that "having at least two cameras mounted on a vehicle and arranged at a predetermined installation distance, and using a plurality of images acquired by the at least two cameras, the present invention is used. In a device that estimates and acquires the three-dimensional coordinates of the target around the vehicle, the three-dimensional coordinates of the target around the vehicle are obtained from two images acquired over time by one of at least two cameras. Monocular stereo processing unit that estimates and calculates, compound-eye stereo processing unit that estimates and calculates the 3D coordinates of the target around the vehicle from two images acquired by at least two cameras at the same time, those monocular The 3D coordinates of the target connected to the stereo processing unit and the compound eye stereo processing unit and calculated and estimated by those processing units are selected and integrated according to a predetermined standard, and estimated as the 3D coordinates of the target to be obtained. It has a result integration / switching means that outputs the result. ”A three-dimensional coordinate acquisition device is disclosed (see summary).

Japanese Unexamined Patent Publication No. 2007-263657

In order to detect the surroundings of a moving object without blind spots using an image sensor, there is a method of installing a drive device in the image sensor and mechanically controlling the image pickup direction to expand the image pickup area in time series. However, since such a drive device deteriorates at an early stage depending on the frequency of use, the replacement cost of the drive device is high.

On the other hand, there is a method of detecting an image of a plurality of regions with a plurality of image sensors. However, the image sensor used in this method has a larger amount of acquired data than the external sensor such as LiDAR, and the communication between the image sensor and the calculation processing unit and the calculation processing load in the calculation processing unit increase. However, this problem can be avoided by adding more computers, reducing the frame rate of the image sensor, compressing the image detected by the image sensor, and thinning out the pixels (reducing the amount of image data). However, as the amount of data processing increases, the number of devices to be installed increases. Therefore, the size and design of the moving body are restricted, and further, the detection performance of the image sensor is deteriorated.

Further, there is a method of estimating the distance of each pixel in one image captured by a monocular by CNN (Convolutional Neural Network) using a plurality of images and distance data prepared in advance as teacher data. Here, the pixel distance is the distance from the camera of the object reflected in the pixel. Although this method can reduce the number of image sensors mounted, it has a drawback that it is not versatile because it estimates only the distance within the range of the distance data included in the teacher data. Therefore, the estimated distance error and accuracy range are not guaranteed, and the processing load is high. Therefore, there is a problem that it cannot be applied in an environment where distance detection is required in real time and with high accuracy such as a human activity environment.

The present invention has been made in view of such a background, and it is an object of the present invention to provide an autonomous mobile device and an autonomous mobile system having a low processing load.

In order to solve the above-mentioned problems, the present invention has at least two first image pickup units having a direction in the positive direction with respect to the straight-ahead direction of the moving body in the image pickup region, with respect to the straight-ahead direction of the moving body. It has at least one second image pickup unit that includes the orientation in the orthogonal direction in the image pickup area, and is in the image pickup area based on the image pickup images captured by the first image pickup unit and the second image pickup unit. A distance calculation unit that calculates the distance to the object, an autonomous movement control unit that calculates the self-position based on the calculated distance, and performs autonomous movement based on the calculated self-position, and the above-mentioned It is an autonomous moving device as the moving body having a state determination unit for determining whether the moving body is stopped or moving, and the distance calculation unit is imaged by the first imaging unit. The distance is calculated based on the disparity of the images, and the distance is calculated based on a plurality of captured images with different imaging times captured by the second imaging unit , and the autonomous movement control unit uses the autonomous movement control unit to calculate the distance. When the moving body moves from a stop to a movement, the moving body is moved in a straight direction for a predetermined distance .
Other solutions will be described as appropriate in the embodiments.

According to the present invention, it is possible to provide an autonomous mobile device and an autonomous mobile system having a low processing load.

It is a figure which shows the structure of the moving body 1 which concerns on 1st Embodiment. It is a figure (the 1) which shows the detail of the distance calculation by a monocular distance calculation unit 113, and the distance error. It is a figure (the 2) which shows the detail of the distance calculation by a monocular distance calculation unit 113, and the distance error. FIG. 3 is a diagram (No. 3) showing details of distance calculation by the monocular distance calculation unit 113 and a distance error. It is a figure which shows the method of the movement control in the moving body 1. It is a figure which shows the structure of the calculation processing part 110a of the moving body 1 by 2nd Embodiment. It is a figure which shows the configuration example of the autonomous movement system Z which concerns on 3rd Embodiment. It is a figure which shows the configuration example of the server 5 which concerns on 3rd Embodiment.

Next, an embodiment (referred to as “embodiment”) for carrying out the present invention will be described in detail with reference to the drawings as appropriate.

[First Embodiment]
First, the first embodiment of the present invention will be described with reference to FIGS. 1 to 3.
FIG. 1 is a diagram showing a configuration of a mobile body 1 according to the first embodiment.
In the present embodiment, an unmanned vehicle equipped with a plurality of image sensors 120, a moving body (autonomous moving device) 1 such as an automobile, a robot, or a ship autonomously moves.
The mobile body 1 includes a plurality of (six in the example of FIG. 1) image sensors (imaging units) 120, a calculation processing unit 110, and a plurality of drive units 130. The image sensor 120 is specifically a camera. Here, as the image sensor 120, a low-resolution and inexpensive camera such as a Web camera can be used.

Of the image sensors 120, the image sensors 120A and 120B are installed so as to include the direction in the positive direction with respect to the moving direction of the moving body 1 in the image pickup region. That is, the image sensors 120A and 120B take an image of the front of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 shown in FIG.
The image sensors 120C and 120D are installed so as to include the direction in the negative direction with respect to the moving direction of the moving body 1 in the image pickup region. That is, the image sensors 120C and 120D take an image of the rear of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 shown in FIG.

Further, the image sensors 120E and 120F are installed so as to include the direction orthogonal to the moving direction of the moving body 1 in the imaging region. That is, the image sensors 120E and 120F take an image of the side of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 shown in FIG. Incidentally, when the moving body 1 moves in the direction of the arrow A1 shown in FIG. 1, the image sensor 120E images the left side of the moving body 1, and the image sensor 120F images the right side of the moving body 1.
As shown in FIG. 1, the image sensors 120E and 120F are installed so as to substantially face each other.
Here, by setting the angle of view of each image sensor 120 to be wide-angle or ultra-wide-angle, it is possible to capture an object around the moving body 1 without fail.

The light receiving band of the image sensor 120 may be determined according to the traveling environment. For example, when traveling indoors or outdoors in the daytime, a camera that receives a visible light band is mounted as an image sensor 120. Further, when traveling outdoors in a dark place or at night, an image sensor 120 that receives light in the near infrared band is mounted.

The frame rate of the image sensor 120 may be set based on the maximum moving speed of the moving body 1, the traveling environment, the braking distance, the maximum moving distance between one frame, and the like. For example, when traveling on an indoor tile at a maximum speed of 6 km / h, assuming that the allowable moving distance when suddenly stopped based on the surrounding captured image is 0.5 m, braking on a tile with a dynamic friction coefficient of 0.3 Considering the distance of 0.47 m, the frame rate of the image sensor 120 is set to about 6000/3600 / (0.5-0.47) = about 56 fps. In an environment where the density of people and obstacles is low, the maximum moving distance between one frame is set to 0.05 m, and for example, the frame rate of the image sensor 120 is set to about 6000/3600/0.05 = about 33 fps. Similarly, when traveling outdoors at a maximum speed of 60 km / h, the maximum moving distance between one frame is 0.05 m, and the frame rate of the image sensor 120 is, for example, about 60000/3600/0.05 = about 333 fps. Is set to.

In an environment where the moving speed is not restricted, the maximum moving speed of the moving body 1 can be determined based on the maximum frame rate of the image sensor 120 that satisfies the predetermined angle of view and the light receiving band, the braking distance, and the like. For example, when the maximum frame rate of the image sensor 120 is 30 fps, the maximum moving distance between one frame is 0.05 m, and the maximum moving speed of the moving body 1 is 0.05 × 30 = 1.5 m / s = 5. It is set to 4 km / h.

The drive unit 130 is a wheel. Then, among the drive units 130, the front drive units 130A and 130B control the direction of the wheel shaft. Further, the rear drive units 130C and 130D control the rotation speed of the wheels. The rear drive units 130C and 130D can be controlled at different rotation speeds. For example, in the case of making a super-credit turn with the center of the line segment connecting the rear drive unit 130C and the rear drive unit 130D as the rotation center, the wheel axes of the front drive units 130A and 130B are oriented in a direction perpendicular to the rotation direction. , The rear drive units 130C and 130D are rotated in opposite directions to each other.

Further, the front drive units 130A and 130B may be casters or the like. In this case, the front drive units 130A and 130B are not controlled. The functions of the front drive units 130A and 130B and the rear drive units 130C and 130D may be interchanged. That is, by controlling the rotation speed of the front drive units 130A and 130B and the direction of the wheel shafts of the rear drive units 130C and 130D, the center of the line segment connecting the front drive unit 130A and the front drive unit 130B is swiveled. You may make a super-credit turn as the center. In addition, it may be a simple turn instead of a super-credit turn.

The calculation processing unit 110 includes a CPU (Central Processing Unit) (not shown), a ROM (Read Only Memory) (not shown), and a RAM (Random Access Memory) (not shown). The program stored in the ROM or the like is loaded into the RAM, and the loaded program is executed by the CPU. By doing so, the data acquisition unit 111, the stereo distance calculation unit 112, the monocular distance calculation unit 113, the object detection unit 114, the self-position calculation unit 115, and the control unit 116 are embodied.

The data acquisition unit 111 acquires the image captured by each image sensor 120 at a set frame rate. Image acquisition is performed synchronously in each image sensor 120.
The stereo distance calculation unit (distance calculation unit) 112 is an object included in both the imaging regions of the image sensors 120A and 120B based on the disparity of the images of the image sensors 120A and 120B acquired at the same time by the data acquisition unit 111. Calculate the distance to. Then, the stereo distance calculation unit 112 applies the optical flow estimation method to a plurality of images with different imaging times captured by at least one of the image sensors 120A and 120B, and the object existing in front of the moving body 1 Track things. Tracking here means detecting the same object in different frames (images at different imaging times).

The stereo distance calculation unit 112 executes the same processing for the image sensors 120C and 120D, and calculates the distance to the object included in the image pickup region of the image sensors 120C and 120D. Further, the stereo distance calculation unit 112 applies an optical flow estimation method to track an object existing behind the moving body 1.

Here, the object is a person to be avoided, an obstacle, a feature point used for calculating the position and posture of the moving body 1 by the self-position calculation unit 115 described later, a landmark, or the like. As the optical flow estimation method described above, for example, the Lucas-Kanade method, the Horn-Schunkk method, or the like is used. Based on the frame rate of the image sensor 120 and the maximum moving speed of the moving body 1, the user determines in advance which method to use as the optical flow estimation method.

The monocular distance calculation unit (distance calculation unit) 113 includes images with different imaging times captured by the image sensors 120E and 120F acquired by the data acquisition unit 111, and the self-position calculation unit 115 calculated later. Based on the time-series information of the position of the moving body 1 at the imaging time, the distance to the object included in each imaging region of the image sensors 120E and 120F is calculated.

Further, the monocular distance calculation unit 113 performs the same processing on any one of the image sensors 120A and 120B, and the distance to the object included in the image pickup region included in any one of the image sensors 120A and 120B. Is calculated. Further, the monocular distance calculation unit 113 performs the same processing on any one of the image sensors 120C and 120D, and the distance to the object included in the image pickup region included in any one of the image sensors 120C and 120D. Is calculated. In short, the monocular distance calculation unit 113 uses the parallax between two images taken at different times to geometrically calculate the distance as if it were compound-eye.

Whether the images captured by the image sensors 120A and 120B and the image sensors 120C and 120D are processed by the stereo distance calculation unit 112 or the monocular distance calculation unit 113 is switched according to the criteria described later.

The image sensor 120A and the image sensor 120C may be installed so that their installation positions are displaced from each other as shown in FIG. 1, or they may be installed so that their positions are aligned (opposed to each other). Similarly, the image sensor 120B and the image sensor 120D may be installed so as to be displaced from each other as shown in FIG. 1, or may be installed so as to be aligned (opposed). Further, the image sensor 120E and the image sensor 120F may be installed so as to be displaced from each other as shown in FIG. 1, or may be installed so as to be aligned (opposed).

The object detection unit (autonomous movement control unit) 114 is based on the distance to each object calculated by the stereo distance calculation unit 112 or the monocular distance calculation unit 113, and the mounting position and posture of each of the image sensors 120. Calculate the position of each object. Then, when the object is a moving object, the object detecting unit 114 calculates the amount of movement of each object per unit time. The amount of movement is calculated based on the positions and times at a plurality of different times in the object tracked by the optical flow estimation method by the stereo distance calculation unit 112 or the monocular distance calculation unit 113, respectively.

The self-position calculation unit (autonomous movement control unit) 115 calculates the position and posture of the moving body 1 at the time when the image is acquired by the data acquisition unit 111. The position and posture are calculated by associating a map created in advance with an object whose position has been calculated by the object detection unit 114 (map matching). Further, the self-position calculation unit 115 corrects the position of the object calculated by the object detection unit 114 based on the calculated position and posture of the moving body 1. That is, the self-position calculation unit 115 corrects the position (distance) of the object calculated by the stereo distance calculation unit 112 and the monocular distance calculation unit 113 from the position / posture of the moving body 1 and the map.

When the object is a moving object, the self-position calculation unit 115 corrects the amount of movement of the object per unit time based on the calculated position and posture of the moving body 1. The map can be obtained by imaging the traveling environment while manually traveling the moving body 1. Then, a map is created by acquiring a plurality of images having different imaging times and applying SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) to the acquired images. Imaging for creating a map is performed, for example, while manually moving a dolly equipped with a plurality of image sensors 120 or by holding and moving a unit equipped with a plurality of image sensors 120.

The following methods exist as methods for associating a map with an object. Any method may be used.
-ICP (Iterative Closest Point) that associates based on the arrangement of feature points.
-SHOT (Signature of Histograms of OrienTations).
-A method of associating feature points with similar feature quantities (brightness gradient vector near the feature points on the image, distribution of the brightness gradient vector of the entire image, etc.).
Which of these methods is adopted is determined by the composition of the map. For example, when the map is composed only of the position information of the feature points, ICP is adopted if the map is two-dimensional, and SHOT is adopted if the map is three-dimensional. When the map is composed of the position information of the feature points and the feature amount, the feature amount is calculated by the same method as the method used at the time of creating the map in the self-position estimation at the time of autonomous movement of the moving body 1. The mapping is performed using.

The control unit (autonomous movement control unit) 116 determines the position and posture of the moving body 1 calculated by the self-position calculation unit 115, the corrected position of the object, and when the object is a moving object, per unit time of the object. The next movement amount of the moving body 1 is determined based on the movement amount of. Then, the control unit 116 calculates the control value to the drive unit 130 based on the next movement amount, and transmits the calculated control value to the drive unit 130.
The arrow A2 will be described later.

Next, with reference to FIGS. 2A to 2C, an error caused by the monocular distance calculation in the moving direction, the lateral direction, and the super-credit turning will be described.
(Rejection error due to monocular distance calculation)
2A to 2C show the details of the distance calculation by the monocular distance calculation unit 113 and the distance error.
In FIG. 2A, it is assumed that the moving body 1 is moving in the direction of the arrow A1 in FIG. Then, FIG. 2A shows an error that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the front image captured by the image sensor 120A or the image sensor 120B.
In FIG. 2A, reference numeral 201 is the position and orientation of the moving body 1 at time t1. At this time, the image sensor 120A takes an image of the object J in front of the moving body 1 with the line of sight 202.

After that, the moving body 1 moves to the point of reference numeral 211 at time t2, and the image sensor 120A images the object J with the line of sight 212.
The monocular distance calculation unit 113 is a distance to the object J based on the position and posture (reference numeral 201) at time t1, the position and posture (reference numeral 211) at time t2, the line of sight 202 and the line of sight 212 at each time. (Line length of line of sight 212) is calculated.
The parallax between the line of sight 202 and the line of sight 212 is calculated by detecting the same point included in the object J from the image at time t1 and the image at time t2. The detection of the same point is performed by using an optical flow estimation method. The position and posture (reference numeral 211) at time t2 are the control values for the drive unit 130 determined by the control unit 116 at time t1 at the position and posture (reference numeral 201) at time t1 calculated by the self-position calculation unit 115 described later. It is a provisional value of the position and posture calculated by integrating. The provisional values of the position and the posture (reference numeral 211) at the time t2 are processed and corrected by the self-position calculation unit 115. That is, the self-position calculation unit 115 corrects the position / posture of the moving body 1 at time t2 by using the map.

Here, it is assumed that the line-of-sight 212 at time t2 images the object J as the line-of-sight 221 due to the sensor error of the image sensor 120A. In this case, the object J is calculated to exist at the point Ja. Therefore, an error E1 occurs in the distance to the object J.
When a rear image captured by either one of the image sensors 120C and 120D is used, the same degree of error occurs.

If the distance between the object J existing in the moving direction of the moving body 1 and the moving body 1 is calculated by monocular distance calculation in this way, a large error will occur.

In FIG. 2B, it is assumed that the moving body 1 is moving in the direction of the arrow A1 in FIG. Then, FIG. 2B shows an error that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the lateral image captured by either the image sensor 120E or the image sensor 120F. ing.
The method of calculating the distance is the same as in the case of FIG. 2A. That is, the object J is imaged from the point of reference numeral 301 at time t1 with the line of sight 302. Then, at time t2, the object J is imaged from the point of reference numeral 311 with the line of sight 312. Then, the distance to the object J (the line segment length of the line of sight 312) is calculated. Here, assuming that the line of sight 312 at time t2 images the object J as the line of sight 321 due to the sensor error of the image sensor 120E, it is calculated that the object J exists at the point Jb. That is, an error E2 occurs in the distance to the object J.

The error angle θ1 shown in FIG. 2A and the error angle θ2 shown in FIG. 2B are almost the same. Rather, the error angle θ2 shown in FIG. 2B is slightly larger than the error angle θ1 shown in FIG. 2A. Nevertheless, when the error E1 shown in FIG. 2A and the error E2 shown in FIG. 2B are compared, the error E2 shown in FIG. 2B is smaller. This is because the distance to the object J is inversely proportional to the parallax between the image at time t1 and the image at time t2. In short, if the parallax between the image at time t1 and the image at time t2 is small, the error tends to be large. On the contrary, if the parallax between the image at time t1 and the image at time t2 is large, the error is unlikely to be large.
Therefore, in order to reduce the error, it is necessary to increase the parallax between the two images as much as possible.

Therefore, it is desirable to apply the processing of the monocular distance calculation unit 113, which has a lower processing load than the stereo distance calculation unit 112, to the imaging region in the direction orthogonal to the moving direction.
As described above, it is desirable that the distance of the image captured by the image sensor 120 provided in the direction orthogonal to the moving direction of the moving body 1 is calculated by the monocular distance calculating unit 113. For example, if the drive unit 130 is an omni wheel or the like, the moving body 1 can move in the direction of the arrow A2 in FIG. When the moving body 1 moves in the direction of the arrow A2 in FIG. 1, the image sensors 120A to 120D are the image sensors 120 installed in the direction orthogonal to the moving direction. In such a case, it is preferable that the monocular distance calculation unit 113 calculates the distance of the images captured by the image sensors 120A to 120D. However, in this case, an image captured by either one of the image sensors 120A and 120B and one of the image sensors 120C and 120D may be used.

Further, the image captured by the image sensor 120 installed in the positive direction (that is, forward) and the negative direction (that is, backward) with respect to the moving direction of the moving body 1 is processed by the stereo distance calculation unit 112.
That is, as shown in FIG. 2A, the error E1 is a large value, so that the images captured by the image sensors 120A and 120B installed in the moving direction of the moving body 1 as shown in FIG. 2A are the monocular distance calculation unit. From 113, it is better to be processed by the stereo distance calculation unit 112.
As described above, the feature of the present application is that even with the same image sensor 120, it is possible to switch between monocular distance calculation and stereo distance calculation depending on the moving direction of the moving body 1.

In FIG. 2C, a case where the moving body 1 is making a super-credit turn will be described.
When the moving body 1 makes a super-credit turn in the direction of the turning direction (arrow A3) of FIG. 2C, it can be said that the moving body 1 is moving in a direction orthogonal to the moving direction shown by the arrow A1 of FIG. That is, in this case, the image sensors 120A to 120D are image sensors 120 (image sensors 120 installed on the side) existing in a direction orthogonal to the moving direction of the moving body 1.

That is, it is better not to apply the processing of the stereo distance calculation unit 112 to the image sensors 120A to 120D while the moving body 1 is making a super-credit turn. That is, when the moving body 1 is making a super-credit turn, the processing of the monocular distance calculation unit 113 is applied to the image captured by at least one of the image sensors 120A and 120B. Similarly, the process of the monocular distance calculation unit 113 is applied to the image captured by at least one of the image sensors 120C and 120D. The reason is described below.

In FIG. 2C, it is assumed that the moving body 1 is turning in the direction of the arrow A3 in FIG. 2C (super-credit turning). Then, FIG. 2C shows an error that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the image captured by the image sensor 120A or the image sensor 120B.

The method of calculating the distance in FIG. 2C is the same as that of FIGS. 2A and 2B. At time t1, the image sensor 120A images the object J with the line of sight 402 at the position / posture of reference numeral 401. After that, the moving body 1 makes a super-credit turn in the super-credit turn direction (arrow A3). Then, at time t2, the image sensor 120A images the object J with the line of sight 412 at the position / posture of the reference numeral 411.

After that, the monocular distance calculation unit 113 calculates the distance to the object J (the line segment length of the line of sight 412) based on the image captured at time t1 and the image captured at time t2. Whether or not the super-credit turning is in progress is determined from the control value of the control unit 116 at time t1.

Here, it is assumed that the line-of-sight 412 at time t2 is the line-of-sight 421 due to the sensor error of the image sensor 120A, and the object J is imaged. In this case, it is calculated that the object J exists at the point Jc, and an error E3 occurs in the distance to the object J.

Here, when the error E1 shown in FIG. 2A and the error E3 shown in FIG. 2C are compared, the error E3 is smaller. This is because the amount of movement in the direction orthogonal to the moving direction, that is, the turning direction (arrow A3) has increased, and the parallax has increased. In other words, in FIG. 2C, since the moving direction of the moving body 1 is the turning direction (arrow A3), the image sensor 120A (or the image sensor 120B) becomes a side image sensor that images the side of the moving body 1. This is because of the fact.

Therefore, the super-credit turning amount is calculated based on the control value of the control unit 116, and when the super-credit turning amount is larger than the preset determination threshold value, an image is taken with at least one of the image sensors 120A and 120B. The processing of the monocular distance calculation unit 113 is applied to the image. In this case, the stereo distance calculation unit 112 does nothing. The same applies to the image sensors 120C and 120D.

As described above, when the image sensor 120 is provided in the direction orthogonal to the moving direction of the moving body 1, the error can be made smaller in the monocular distance calculation than in the stereo distance calculation. Further, in the monocular distance calculation, the processing data can be reduced by the stereo distance calculation, so that the processing load can be reduced.
That is, when the moving body 1 is making a super-credit turn, a method of calculating the distance by the monocular distance calculation unit 113 rather than calculating the distance by the stereo distance calculation unit 112 from the images captured by the image sensors 120A and 120B. Can reduce the error. Further, by calculating the distance by the monocular distance calculation unit 113, the amount of image data to be processed can be reduced and the processing load can be reduced.

Which of the image sensors 120A and 120B the image captured by the image sensor 120 is applied to the monocular distance calculation unit 113 is determined based on the amount of movement of each image sensor 120 due to the super-credit turning. For example, among the image sensors 120A and 120B, if the amount of movement of the image sensor 120A is large, the image captured by the image sensor 120A is used. The same applies to the image sensors 120C and 120D.
However, when the image sensors 120A and 120B are in a symmetrical positional relationship with respect to the central axis of the moving body 1, the amount of movement due to the super-credit turning of each of the image sensors 120A and 120B is the same. In such a case, the image of the image sensor 120 having many feature points on the captured image is used. The same applies to the image sensors 120C and 120D.

The determination thresholds for calculating the distance by the monocular distance calculation unit 113 or the stereo distance calculation unit 112 are preset in the image sensors 120A and 120B and the image sensors 120C and 120D, respectively. For example, the distance (baseline distance) between the image sensor 120A and the image sensor 120B is set as a determination threshold value. The same applies to the image sensors 120C and 120D. Here, the baseline distance is the parallax in the calculation of the stereo distance. That is, when images are captured at different times, the distance traveled by the moving body 1 in the direction of the arrow A2 in FIG. 1 between the respective imaging times is larger than the baseline distance, whichever is the image sensor 120A or 120B. On the other hand, the monocular distance calculation is applied to the captured image. When the distance traveled by the moving body 1 in the direction of the arrow A2 in FIG. 1 between the respective imaging times is smaller than the baseline distance, the stereo distance is calculated for the images captured by the image sensor 120A and the image sensor 120B. Applies. The same applies to the image sensors 120C and 120D.

Incidentally, since the image sensors 120E and 120F are provided in the direction orthogonal to the moving direction of the moving body 1 even during the super-credit turning, the distance calculation by the monocular distance calculating unit 113 is applied.

Since CNN is an estimated distance by learning, the accuracy of the distance is not guaranteed in the monocular distance calculation using CNN. In the present embodiment, since the distance is geometrically obtained, the accuracy of the distance calculated from the CNN is guaranteed, although an error occurs.

In addition, although the super-credit turning is described here, it can also be applied to a curve. When the moving body 1 is curved, when the vector of the moving direction of the moving body 1 is decomposed into the vector of the straight direction and the vector of the direction orthogonal to the straight direction (orthogonal direction), the straight direction component is larger than the orthogonal direction component. , The stereo distance calculation is applied to the image captured by either one of the image sensors 120A and 120B. When the orthogonal direction component is larger than the straight direction component, the monocular distance calculation is applied to the image captured by either one of the image sensors 120A and 120B. The same applies to the image sensors 120C and 120D.

(Movement control)
FIG. 3 is a diagram showing a method of movement control in the moving body 1.
First, a target route W to the destination point is created based on the object J1 such as a wall created in advance and the destination point position. As a method for creating the target route W, for example, there are the Dijkstra method and the A-Star method. In this method, the starting point and the ending point are first specified. Then, a potential field P is provided in the vicinity of each feature point constituting the object J1. The potential field P is a region avoided by the moving body 1. Next, the control unit 116 performs a route search with the minimum cost based on the distance to the destination point and the potential field P. Then, in the control unit 116, the route having the lowest cost is generated as the target route W. The target route W is generated by, for example, a server (not shown) and then transmitted to the mobile body 1.

Next, the control unit 116 sets a plurality of candidates B for the next moving speed and turning angular velocity around the moving speed and turning angular velocity B1 of the moving body 1 calculated based on the previous control value. Then, the control unit 116 calculates the candidate position of the next moving body 1 when each candidate B is adopted.

Each moving speed and turning angular velocity in each candidate B are set so as not to exceed the maximum moving speed and the maximum turning angular velocity of the moving body 1. For example, it is assumed that the moving speed of the candidate B1 is Vm and the moving speed of the candidate B2 is set in the range of (Vm-3km / h) to (Vm + 3km / h). At this time, if the maximum moving speed is (Vm + 2km / h), the moving speed candidate B2 is reset to the range from (Vm-3km / h) to (Vm + 2km / h).

Next, the control unit 116 has a potential field P1 in the vicinity of the moving objects M1 and M2 based on the positions of the moving objects M1 and M2 corrected by the self-position calculation unit 115 and the moving amounts M11 and M21 per unit time. , P2 is provided. Then, the control unit 116 compares the cost from the next movement destination candidate position of the moving body 1 to the target route W with the cost from the next moving destination candidate position of the moving body 1 to the target point (movement end point). At this time, the control unit 116 calculates the cost by a path avoiding each potential field P, P1, P2.
Then, the control unit 116 selects the candidate B having the lowest cost. Then, the control unit 116 adopts the candidate B, which is the minimum cost, as the next moving speed and turning angular velocity. After that, the control unit 116 calculates a control value for the drive unit 130 based on the adopted moving speed and turning angular velocity.

In the example of FIG. 3, in order to avoid the moving objects M1 and M2, the moving body 1 has already deviated from the target path W.

In the first embodiment, when the image sensor 120 is installed in the moving direction of the moving body 1, the stereo distance calculation is applied. Further, when the image sensor 120 is installed in the direction orthogonal to the moving direction, the monocular distance calculation is applied. By doing so, the moving body 1 selects the optimum distance calculation method so that the processing load is reduced according to the moving / turning direction. As a result, it is possible to realize autonomous movement with high accuracy and low processing load while having no blind spot. Further, by selecting a distance calculation method that reduces the error, it is possible to prevent the accuracy of the calculated distance from being lowered even with an inexpensive camera.
As a result, it is possible to prevent the obstacle detection accuracy from being lowered even with an inexpensive camera.
The technique described in Patent Document 1 monocularly views a portion where the fields of view of the camera do not overlap, regardless of the moving direction of the vehicle. On the other hand, the present embodiment is different in that the stereo distance calculation and the monocular distance calculation are switched according to the moving direction of the moving body 1.

Further, until now, the frequency of image acquisition has been reduced to reduce the processing load, but such a method may cause a decrease in performance such as a decrease in distance calculation accuracy. In the present embodiment, it is possible to prevent the distance calculation accuracy from being lowered without reducing the image acquisition frequency.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG.
In the second embodiment, the mobile body 1 safely starts / resumes autonomous movement at the time of activation, when the vehicle is stopped after arriving at the destination, when the vehicle is stagnant due to crowds, or after being temporarily stopped when boarding an elevator or the like. With the goal.

FIG. 4 is a diagram showing a configuration of a calculation processing unit 110a of the mobile body 1 according to the second embodiment.
The calculation processing unit 110a has a state determination unit 117 and a turning angular velocity changing unit (autonomous movement control unit) 118 in addition to the units 111 to 116 having the same configuration as that of the first embodiment.
Each part 111 to 116 autonomously moves the moving body 1 by executing the same processing as each part 111 to 116 in the first embodiment.
The state determination unit 117 determines the traveling state of the moving body 1 based on the time-series information of the position and posture of the moving body 1 calculated by the self-position calculation unit 115 and the activation information. When the calculation processing unit 110a is started, or when both the position and the posture of the moving body 1 have not changed in the past fixed period (for example, 30 seconds) including the current time, the state determination unit 117 sets the running state to ". Judged as "stop / stagnation". In other cases, the state determination unit 117 determines the traveling state as "traveling".

The turning angular velocity changing unit 118 sets the maximum turning angular velocity of the moving body 1 based on the traveling state determined by the state determination unit 117. If the running condition is "stop / stagnation", set the maximum turning angular velocity to zero. This is because the monocular distance calculation cannot be calculated unless the images are acquired in chronological order. That is, when the traveling state is "stop / stagnation", the distance of the object cannot be calculated by the image sensors 120E and 120F. As a result, the distance of the object existing on the side of the moving body 1 cannot be calculated, so that the side of the moving body 1 is a blind spot. Therefore, if the moving body 1 is turned and started, it may collide with an unexpected obstacle. Therefore, the maximum turning angular velocity is set to zero, and the vehicle can only start in the straight direction.

When the running state is "running", the distance between the position of the moving body 1 at the time when the running state is switched from "stop / stagnation" to "running" and the current position of the moving body 1 is a fixed amount (for example, 1 m). When it exceeds, the maximum turning angular velocity is set to the original value of the moving body 1. This is because, as described above, the images need to be aligned to some extent in time series in order to calculate the monocular distance.

The moving body 1 of the second embodiment controls the drive unit 130 so that it does not turn until it goes straight or backwards by a certain amount when it starts or restarts running from the time of starting or when it is stopped or stagnated. By doing so, until the distance to the object existing on the side of the moving body 1 can be accurately calculated, the object existing in front of the moving body 1 and the object existing in the rear are reached. Move while calculating the distance by stereo distance calculation. By doing so, the moving body 1 can be moved in a safe direction.

For example, the determination of whether or not the position and posture of the moving body 1 has changed is performed by the following procedure. First, the state determination unit 117 calculates the variance between the position and the posture in the past fixed period including the current time. Then, the state determination unit 117 determines whether or not the position and posture of the moving body 1 have changed by comparing the preset state determination threshold value with the calculated variance.

Here, if the state determination threshold is set small, it is determined that the variance is stopped near 0. Further, when the state determination threshold value is set to be large, it is determined that the moving body 1 is stopped even when it is moving considerably.
If the state determination threshold is set small, it is determined that the moving body 1 is moving even if it moves only a little. Therefore, before the image sensor 120 installed on the side of the moving body 1 moves the distance required for calculating the monocular distance, the monocular distance is calculated by the image sensor 120 installed on the side.
On the contrary, when the state determination threshold value is set to a large value, it is determined that the moving body 1 is stopped even when the moving body 1 is considerably moving, as described above. That is, after the image sensor 120 installed on the side of the moving body 1 moves the distance required for calculating the monocular distance, the monocular distance is calculated by the image sensor 120 installed on the side.

From such a feature, when there are many crowds or obstacles in the surroundings, it is preferable to set a large state determination threshold value, and in a place where there is nothing in the surrounding area, it is preferable to set a small state determination threshold value. By doing so, it is possible to realize efficient movement of the moving body 1.

[Third Embodiment]
FIG. 5 is a diagram showing a configuration example of the autonomous mobile system Z according to the third embodiment.
As shown in FIG. 5, the autonomous mobile system Z has a mobile body 1b and a server 5.
The mobile body 1b transmits the image captured by each image sensor 120 to the server 5. The server 5 calculates the distance from the moving object 1b to the object by using the monocular distance calculation or the stereo distance calculation based on the transmitted image. Then, the server 5 calculates the position of the object based on the calculated distance to the object, and further calculates the self-position (position / posture) of the moving body 1b by map matching. Then, the server 5 calculates the next movement speed and the turning angle of the moving body 1b based on the self-position of the moving body 1b and the like. After that, the server 5 transmits the self-position of the moving body 1b and the next moving speed and turning angle of the moving body 1b to the moving body 1b as movement instruction information. The moving body 1b moves according to the received movement instruction information. Wireless communication is preferable for communication between the mobile body 1b and the server 5.
In addition to the above-mentioned processing, the server 5 also performs processing performed by the stereo distance calculation unit 112, the monocular distance calculation unit 113, the object detection unit 114, the self-position calculation unit 115, and the control unit 116 in the first embodiment. Do as appropriate.

The calculation processing unit 110b of the mobile body 1b includes a data transmission unit 119a, a data reception unit 119b, and a movement control unit 119c.
The data transmission unit 119a transmits the image captured by each image sensor 120 to the server 5.
The data receiving unit 119b receives the movement instruction information from the server 5.
Further, the movement control unit 119c controls each drive unit 130 based on the movement instruction information sent from the server 5.
Since the image sensors 120 (120A to 120F) and the drive units 130 (130A to 130D) are the same as those in FIG. 1, the description thereof is omitted here.
The details of the server 5 will be described with reference to FIG.

FIG. 6 is a diagram showing a configuration example of the server 5 according to the third embodiment.
The server 5 includes a data receiving unit 511, a stereo distance calculation unit 512, a monocular distance calculation unit 513, an object detection unit 514, a self-position calculation unit 515, a control unit 516, and a data transmission unit 517.
The data receiving unit 511 receives the images captured by the respective image sensors 120 from the moving body 1b.
The stereo distance calculation unit 512, the monocular distance calculation unit 513, the object detection unit 514, and the self-position calculation unit 515 are the stereo distance calculation unit 112, the monocular distance calculation unit 113, the object detection unit 114, and the self-position calculation unit in FIG. Since the operation is the same as that of 115, the description thereof is omitted here.
The control unit 516 performs operations other than the control of the drive unit 130 among the operations performed by the control unit 216 in FIG. 1.
The data transmission unit 517 includes a stereo distance calculation unit 512, a monocular distance calculation unit 513, an object detection unit 514, a self-position calculation unit 515, a self-position of the moving body 1b calculated by processing by the control unit 516, and a moving body. The next movement speed / turning angle of 1b is transmitted to the moving body 1b as movement instruction information.

The server 5 includes a CPU (not shown), a RAM (not shown), an HD (Hard Disk) (not shown), and the like. Then, the program stored in the HD or the like is loaded into the RAM, and the loaded program is executed by the CPU. By doing so, the data receiving unit 511, the stereo distance calculation unit 512, the monocular distance calculation unit 513, the object detection unit 514, the self-position calculation unit 515, and the control unit 516 are embodied.

According to the third embodiment, since the calculation of the distance required for the movement of the moving body 1b and the position calculation of the moving body 1b are performed by the server 5, the processing load of the moving body 1b can be reduced. Incidentally, the server 5 can calculate the distance and the position of the moving body 1b for the plurality of moving bodies 1b.
Further, at least one of the stereo distance calculation unit 512, the monocular distance calculation unit 513, the object detection unit 514, the self-position calculation unit 515, and the control unit 516 constituting the server 5 shown in FIG. 6 is provided in the moving body 1b. You may.

For example, an omni wheel can be used as the drive unit 130. In such a case, the moving body 1 can move in the direction of the image sensor 120E and the image sensor 120F of FIG. In such a configuration, two image sensors 120 may be provided in the direction of the image sensor 120E or the image sensor 120F.

The mobile body 1 of the present embodiment can be applied to an autonomous mobile device such as a robot, a vehicle that autonomously moves, a personal mobility that autonomously moves, and the like.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each configuration, function, each unit 111 to 118, a storage unit, etc. described above may be realized by hardware, for example, by designing a part or all of them by an integrated circuit or the like. Further, each configuration, function, etc. described above may be realized by software by interpreting and executing a program in which a processor such as a CPU realizes each function. In addition to storing information such as programs, tables, and files that realize each function in HD (Hard Disk), memory, recording devices such as SSD (Solid State Drive), IC (Integrated Circuit) cards, etc. , SD (Secure Digital) card, DVD (Digital Versatile Disc) and other recording media.
Further, in each embodiment, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines are necessarily shown in the product. In practice, you can think of almost all configurations as interconnected.

1 Mobile (autonomous mobile device)
5 Server 112,512 Stereo distance calculation unit (distance calculation unit)
113,513 Monocular distance calculation unit (distance calculation unit)
114,514 Object detection unit (autonomous movement control unit)
115,515 Self-position calculation unit (autonomous movement control unit)
116,516 Control unit (autonomous movement control unit)
117, Status determination unit 118 Turning angular velocity change unit (autonomous movement control unit)
120, 120A-120F image sensor (imaging unit)

Claims (5)

It has at least two first image pickup units that include the direction in the positive direction with respect to the straight direction of the moving body in the image pickup region.
Each has at least one second image pickup unit that includes the direction orthogonal to the straight-ahead direction of the moving body in the image pickup region.
A distance calculation unit that calculates the distance to an object in the image pickup region based on the image captured by the first image pickup unit and the second image pickup unit.
An autonomous movement control unit that calculates a self-position based on the calculated distance and performs autonomous movement based on the calculated self-position .
With a state determination unit that determines whether the moving body is stopped or moving
An autonomous mobile device as the mobile body having the above.
The distance calculation unit
The distance is calculated based on the parallax of the image captured by the first imaging unit.
The distance is calculated based on a plurality of captured images with different imaging times captured by the second imaging unit .
The autonomous movement control unit
When the moving body moves from a stop to a moving body, the moving body is moved in a straight direction for a predetermined distance.
An autonomous mobile device characterized by that. It has at least two first image pickup units that include the direction in the positive direction with respect to the straight direction of the moving body in the image pickup region.
Each has at least one second image pickup unit that includes the direction orthogonal to the straight-ahead direction of the moving body in the image pickup region.
A distance calculation unit that calculates the distance to an object in the image pickup region based on the image captured by the first image pickup unit and the second image pickup unit.
An autonomous movement control unit that calculates a self-position based on the calculated distance and performs autonomous movement based on the calculated self-position .
With a state determination unit that determines whether the moving body is stopped or moving
An autonomous mobile device as the mobile body having the above.
The distance calculation unit
The distance is calculated based on the parallax of the image captured by the first imaging unit.
The distance is calculated based on a plurality of captured images with different imaging times captured by the second imaging unit.
The autonomous movement control unit
When the moving body moves from a stop to a moving body, the moving body is moved in a straight direction for a predetermined distance.
Judgment as to whether the moving body is stopped or moving is performed by comparing the variance of the position of the moving body with the determination threshold value.
The determination threshold is changed depending on the surrounding environment in which the moving object moves.
An autonomous mobile device characterized by that. The first image pickup unit has at least two third image pickup units that include a direction in the negative direction with respect to the straight direction of the moving body in the image pickup region.
The distance calculation unit
The autonomous mobile device according to claim 1 or 2 , wherein the distance is calculated based on the parallax of the image captured by the third imaging unit. Equipped with a mobile and a server,
The moving body is
It has at least two first imaging units including a direction in the positive direction with respect to the straight direction of the moving body in the imaging region.
It has at least one second image pickup unit that includes the direction orthogonal to the straight direction of the moving body in the image pickup region, and at the same time, it has at least one second image pickup unit.
An autonomous movement control unit that performs autonomous movement based on the self-position of the moving body sent from the server .
With a state determination unit that determines whether the moving body is stopped or moving
Have and
The server
A distance calculation unit that calculates the distance to an object in the image pickup region based on the image captured by the first image pickup unit and the second image pickup unit.
A position calculation unit that calculates the self-position of the moving body based on the calculated distance, and
Have,
The distance calculation unit
The distance is calculated based on the parallax of the image captured by the first imaging unit.
The distance is calculated based on a plurality of captured images with different imaging times captured by the second imaging unit .
The autonomous movement control unit
When the moving body moves from a stop to a moving body, the moving body is moved in a straight direction for a predetermined distance.
An autonomous mobile system characterized by that. Equipped with a mobile and a server,
The moving body is
It has at least two first imaging units including a direction in the positive direction with respect to the straight direction of the moving body in the imaging region.
It has at least one second image pickup unit that includes the direction orthogonal to the straight direction of the moving body in the image pickup region, and at the same time, it has at least one second image pickup unit.
An autonomous movement control unit that performs autonomous movement based on the self-position of the moving body sent from the server .
With a state determination unit that determines whether the moving body is stopped or moving
Have and
The server
A distance calculation unit that calculates the distance to an object in the image pickup region based on the image captured by the first image pickup unit and the second image pickup unit.
A position calculation unit that calculates the self-position of the moving body based on the calculated distance, and
Have,
The distance calculation unit
The distance is calculated based on the parallax of the image captured by the first imaging unit.
The distance is calculated based on a plurality of captured images with different imaging times captured by the second imaging unit .
The autonomous movement control unit
When the moving body moves from a stop to a moving body, the moving body is moved in a straight direction for a predetermined distance.
Judgment as to whether the moving body is stopped or moving is performed by comparing the variance of the position of the moving body with the determination threshold value.
The determination threshold is changed depending on the surrounding environment in which the moving object moves.
An autonomous mobile system characterized by that.

Images