WO2019220740A1

WO2019220740A1 - Autonomous movement device and autonomous movement system

Info

Publication number: WO2019220740A1
Application number: PCT/JP2019/007890
Authority: WO
Inventors: 太紀飯村; 一野瀬　亮子
Original assignee: 株式会社日立製作所
Priority date: 2018-05-15
Filing date: 2019-02-28
Publication date: 2019-11-21
Also published as: JP2019200509A; DE112019001454T5; JP7064948B2

Abstract

In order to provide an autonomous movement device and an autonomous movement system having low processing loads, a moving body (1) comprises: at least two image sensors (120) each having an image capturing region including a positive direction and at least two image sensors (120) each having an image capturing region including a negative direction with respect to the straight advancing direction of the moving body 1; and at least one image sensor (120) having an image capturing region including one direction orthogonal to the straight advancing direction of the moving body (1) and at least one image sensor (120) having an image capturing region including the other direction orthogonal to the straight advancing direction, wherein the distance to an object in the image capturing region is calculated on the basis of the image sensors (120) and captured images captured by the image sensors (120), the position of the moving body is calculated on the basis of the calculated distance, and autonomous movement is carried out on the basis of the calculated position of the moving body. The moving body (1) is characterized in that the distance is calculated on the basis of the parallax of the images captured by the image sensors (120), and the distance is calculated on the basis of a plurality of images captured at different image-capturing times by the image sensors (120).

Description

Autonomous mobile device and autonomous mobile system

The present invention relates to a technology for an autonomous mobile device and an autonomous mobile system.

Conventionally, a technology related to autonomous movement has been developed in which an external sensor or an internal sensor is mounted on a moving body such as a robot or an automobile, and autonomously moves to a destination while detecting and avoiding obstacles around the moving body. ing. Here, the external sensor is an image sensor, LiDAR, or the like, and the internal sensor is an encoder, an inertial measurement device, or the like. Among them, in order to reduce the cost of the sensor to be mounted, there is an increasing demand for technological development using a relatively inexpensive image sensor.

For example, Patent Document 1 states that “it has at least two cameras mounted on a vehicle and arranged at a predetermined installation distance, and using a plurality of images acquired by the at least two cameras, In an apparatus for estimating and acquiring the three-dimensional coordinates of a target around the vehicle, the three-dimensional coordinates of the target around the vehicle are obtained from two images acquired over time by at least one of the two cameras. A monocular stereo processing unit that performs estimation calculation, a compound eye stereo processing unit that estimates and calculates the three-dimensional coordinates of a target around the vehicle from two images simultaneously acquired by two of at least two cameras, and these monoculars The three-dimensional coordinates of the target connected to the stereo processing unit and the compound-eye stereo processing unit, respectively calculated and estimated by these processing units, are selected and integrated according to a predetermined criterion, and estimated as the three-dimensional coordinates of the target to be obtained. And that And outputs the result, result integration-switching means, the a "3-dimensional coordinate acquiring device is disclosed (see Abstract).

JP 2007-263657 A

In order to detect the surroundings of a moving body without using a blind spot using an image sensor, there is a technique of installing a driving device in the image sensor and mechanically controlling the imaging direction to expand the imaging area in time series. However, since such a drive device deteriorates at an early stage depending on the frequency of use, the drive device needs to be replaced.

On the other hand, there is a technique for detecting images of a plurality of regions by a plurality of image sensors. However, the image sensor used in this method has a larger amount of acquired data than an external sensor such as LiDAR, and communication between the image sensor and the calculation processing unit and a calculation processing load in the calculation processing unit increase. However, this problem can be avoided by adding computers, reducing the frame rate of the image sensor, compressing the image detected by the image sensor, or thinning out pixels (reducing the amount of image data). However, as the amount of data processing increases, more devices are installed. Therefore, the size and design of the moving body are restricted, and further, the detection performance of the image sensor is deteriorated.

Also, there is a method for estimating the distance of each pixel in one image captured with a single eye by CNN (Convolutional Neural Network) using a plurality of images prepared in advance and distance data as teacher data. Here, the pixel distance is the distance from the camera of the object shown in the pixel. Although this method can reduce the number of mounted image sensors, it has a drawback that it is not versatile because it estimates only the distance within the range of distance data included in the teacher data. Therefore, the estimated distance error and accuracy range are not guaranteed, and the processing load is high. For this reason, there is a problem that it cannot be applied in an environment that requires distance detection with high accuracy in real time such as a human activity environment.

The present invention has been made in view of such a background, and an object of the present invention is to provide an autonomous mobile device and an autonomous mobile system with a low processing load.

In order to solve the above-described problem, the present invention has at least two first imaging units including, in an imaging region, a positive direction with respect to a straight direction of a moving body, and the straight direction of the moving body Each having at least one second imaging unit including an orientation in the orthogonal direction in the imaging region, and based on the captured images captured by the first imaging unit and the second imaging unit. A distance calculation unit that calculates a distance to the target object, and an autonomous movement control unit that calculates a self-position based on the calculated distance and performs autonomous movement based on the calculated self-position. The distance calculating unit calculates the distance based on the parallax of the image captured by the first imaging unit, and generates a plurality of captured images captured by the second imaging unit and having different imaging times. Based on this, the distance is calculated.
Other solutions will be described as appropriate in the embodiments.

According to the present invention, an autonomous mobile device and an autonomous mobile system with a low processing load can be provided.

It is a figure which shows the structure of the mobile body 1 which concerns on 1st Embodiment. FIG. 10 is a diagram (part 1) illustrating details of distance calculation and distance error in a monocular distance calculation unit 113; FIG. 10 is a diagram (part 2) illustrating details of distance calculation and distance error in the monocular distance calculation unit 113; FIG. 11 is a diagram (part 3) illustrating details of distance calculation and distance error in the monocular distance calculation unit 113; FIG. 3 is a diagram illustrating a movement control method in a moving body 1. It is a figure which shows the structure of the calculation process part 110a of the moving body 1 by 2nd Embodiment. It is a figure which shows the structural example of the autonomous mobile system Z which concerns on 3rd Embodiment. It is a figure which shows the structural example of the server 5 which concerns on 3rd Embodiment.

Next, modes for carrying out the present invention (referred to as “embodiments”) will be described in detail with reference to the drawings as appropriate.

[First Embodiment]
First, a first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a diagram illustrating a configuration of a moving body 1 according to the first embodiment.
In the present embodiment, an unmanned vehicle equipped with a plurality of image sensors 120, a car, a robot, and a mobile body (autonomous mobile device) 1 such as a ship autonomously moves.
The moving body 1 includes a plurality (six in the example of FIG. 1) of image sensors (imaging units) 120, a calculation processing unit 110, and a plurality of driving units 130. Specifically, the image sensor 120 is a camera. Here, as the image sensor 120, a low-resolution and inexpensive camera such as a Web camera can be used.

Among the image sensors 120, the image sensors 120 A and 120 B are installed so that the imaging region includes a positive direction with respect to the moving direction of the moving body 1. That is, the

image sensors

120A and 120B image the front of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 shown in FIG.
The image sensors 120 C and 120 D are installed so that the imaging region includes a negative direction with respect to the moving direction of the moving body 1. That is, the

image sensors

120C and 120D capture an image of the rear of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 illustrated in FIG.

In addition, the

image sensors

120E and 120F are installed so that the imaging region includes a direction orthogonal to the moving direction of the moving body 1, respectively. That is, the

image sensors

120E and 120F image the side of the moving body 1 when the moving body 1 moves in the direction of the arrow A1 shown in FIG. Incidentally, when the moving body 1 moves in the direction of the arrow A1 shown in FIG. 1, the image sensor 120E images the left side of the moving body 1, and the image sensor 120F images the right side of the moving body 1.
As shown in FIG. 1, the

image sensors

120E and 120F are installed so as to be substantially opposed to each other.
Here, by setting the angle of view of each image sensor 120 to a wide angle or an ultra-wide angle, it is possible to capture an object around the moving body 1 without any gaps.

The light receiving band of the image sensor 120 may be determined according to the traveling environment. For example, when traveling indoors or outdoors in the daytime, a camera that receives a visible light band is mounted as the image sensor 120. Further, when traveling outdoors in the dark or at night, an image sensor 120 that receives light in the near infrared band is mounted.

The frame rate of the image sensor 120 may be set based on the maximum moving speed of the moving body 1, the traveling environment, the braking distance, the maximum moving distance between frames, and the like. For example, when traveling on an indoor tile at a maximum speed of 6 km / h, if the allowable travel distance is 0.5 m when suddenly stopped based on the surrounding captured image, braking on the tile with a dynamic friction coefficient of 0.3 Considering the distance of 0.47 m, the frame rate of the image sensor 120 is set to about 6000/3600 / (0.5−0.47) = about 56 fps. In an environment where the density of people and obstacles is low, the maximum moving distance between one frame is set to 0.05 m. For example, the frame rate of the image sensor 120 is set to about 6000/3600 / 0.05 = about 33 fps. Similarly, when traveling outdoors at a maximum speed of 60 km / h, the maximum movement distance between one frame is set to 0.05 m, and the frame rate of the image sensor 120 is, for example, about 60000/3600 / 0.05 = about 333 fps. Set to

In an environment where there is no restriction on the moving speed, the maximum moving speed of the moving body 1 can be determined based on the maximum frame rate of the image sensor 120 satisfying the predetermined angle of view and the light receiving band, the braking distance, and the like. For example, when the maximum frame rate of the image sensor 120 is 30 fps, the maximum moving distance between one frame is 0.05 m, and the maximum moving speed of the moving body 1 is 0.05 × 30 = 1.5 m / s = 5. It is set to 4 km / h.

The drive unit 130 is a wheel. And among the drive parts 130, front part drive

part

130A, 130B controls the direction of a wheel shaft. The

rear drive units

130C and 130D control the rotation speed of the wheels. The

rear drive units

130C and 130D can be controlled at different rotational speeds. For example, in the case of super-superior turning with the center of the line segment connecting the rear drive unit 130C and the rear drive unit 130D as the turning center, the wheel shafts of the

front drive units

130A and 130B are directed in a direction perpendicular to the rotation direction Then, the

rear drive units

130C and 130D are rotated in opposite directions.

Further, the

front drive units

130A and 130B may be casters or the like. In this case, the

front drive units

130A and 130B are not controlled. The functions of the

front drive units

130A and 130B and the

rear drive units

130C and 130D may be interchanged. In other words, by controlling the rotational speed of the

front drive units

130A and 130B and the direction of the wheel shafts of the

rear drive units

130C and 130D, the center of the line segment connecting the front drive unit 130A and the front drive unit 130B is turned. You may make a super turn as the center. It should be noted that a simple turn may be used instead of a superstitch turn.

The calculation processing unit 110 includes a CPU (Central Processing Unit) (not shown), a ROM (Read Only Memory) (not shown), and a RAM (Random Access Memory) (not shown). A program stored in the ROM or the like is loaded into the RAM, and the loaded program is executed by the CPU. By doing in this way, the data acquisition part 111, the stereo distance calculation part 112, the monocular distance calculation part 113, the target object detection part 114, the self-position calculation part 115, and the control part 116 are embodied.

The data acquisition unit 111 acquires an image captured by each image sensor 120 at a set frame rate. Image acquisition is performed synchronously in each image sensor 120.
The stereo distance calculation unit (distance calculation unit) 112 is based on the parallax of the images of the

image sensors

120A and 120B acquired at the same time by the data acquisition unit 111, and is included in the imaging regions of both the

image sensors

120A and 120B. The distance to is calculated. Then, the stereo distance calculation unit 112 applies the optical flow estimation method to a plurality of images captured by at least one of the image sensors 120 A and 120 B at different imaging times, and is present in front of the moving object 1. Track things. The tracking here is to detect the same object in different frames (images at different imaging times).

The stereo distance calculation unit 112 performs the same processing for the

image sensors

120C and 120D, and calculates the distance to the target object included in the imaging regions of the

image sensors

120C and 120D. Furthermore, the stereo distance calculation unit 112 applies an optical flow estimation method and tracks an object existing behind the moving object 1.

Here, the object is a person to be avoided, an obstacle, a feature point used for calculating the position and orientation of the moving body 1 by the self-position calculating unit 115 described later, a landmark, or the like. As the above-described optical flow estimation method, for example, the Lucas-Kanade method, the Horn-Schunkck method, or the like is used. Based on the frame rate of the image sensor 120 and the maximum moving speed of the moving body 1, the user determines in advance which method to use as the optical flow estimation method.

The monocular distance calculation unit (distance calculation unit) 113 calculates the images taken by the

image sensors

120E and 120F acquired by the data acquisition unit 111 at different imaging times and the self-position calculation unit 115 described later. Based on the time-series information of the position of the moving body 1 at the imaging time, the distances to the objects included in the imaging regions of the

image sensors

120E and 120F are calculated.

In addition, the monocular distance calculation unit 113 performs the same processing for any one of the

image sensors

120A and 120B, and the distance to the target object included in the imaging region included in any one of the

image sensors

120A and 120B. Is calculated. Furthermore, the monocular distance calculation unit 113 performs the same processing for any one of the

image sensors

120C and 120D, and the distance to the target object included in the imaging region included in any one of the

image sensors

120C and 120D. Is calculated. In short, the monocular distance calculation unit 113 uses the parallax between two images taken at different times to calculate the distance as if it were a compound eye view.

Whether the images captured by the

image sensors

120A and 120B and the

image sensors

120C and 120D are processed by the stereo distance calculation unit 112 or the monocular distance calculation unit 113 is switched according to the criteria described later.

Note that the image sensor 120A and the image sensor 120C may be installed so that their installation positions are shifted as shown in FIG. 1, or may be installed so that their positions are aligned (facing each other). Similarly, the image sensor 120B and the image sensor 120D may be installed so that their installation positions are shifted as shown in FIG. 1, or may be installed so that their positions are aligned (facing each other). Further, the image sensor 120E and the image sensor 120F may be installed so that their installation positions are shifted as shown in FIG. 1, or may be installed so that their positions are aligned (facing each other).

The object detection unit (autonomous movement control unit) 114 is based on the distance to each object calculated by the stereo distance calculation unit 112 or the monocular distance calculation unit 113 and the mounting position and orientation of each of the image sensors 120. The position of each object is calculated. If the object is a moving object, the object detection unit 114 calculates the amount of movement per unit time of each object. The amount of movement is calculated based on the position and time at a plurality of different times on the object tracked by the optical flow estimation method by the stereo distance calculation unit 112 or the monocular distance calculation unit 113, respectively.

The self position calculation unit (autonomous movement control unit) 115 calculates the position and posture of the moving body 1 at the time when the data acquisition unit 111 acquires the image. The position and orientation are calculated by associating a map prepared in advance with the object whose position is calculated by the object detection unit 114 (map matching). In addition, the self-position calculation unit 115 corrects the position of the target object calculated by the target object detection unit 114 based on the calculated position and orientation of the moving body 1. That is, the self-position calculation unit 115 corrects the position (distance) of the target object calculated by the stereo distance calculation unit 112 or the monocular distance calculation unit 113 from the position / posture of the moving body 1 and the map.

Further, when the object is a moving object, the self-position calculating unit 115 corrects the amount of movement of the object per unit time based on the calculated position and orientation of the moving body 1. The map is obtained by capturing an image of the traveling environment while the mobile body 1 is traveling manually. A plurality of images having different imaging times are acquired, and a map is created by applying SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) to the acquired images. Imaging for creating a map is performed, for example, while manually moving a carriage equipped with a plurality of image sensors 120 or while holding a unit equipped with a plurality of image sensors 120 and moving them.

The following methods exist as a method for associating a map with an object. Any method may be used.
ICP (Iterative Closest Point) to be associated based on the arrangement relationship of feature points.
-SHOT (Signature of Histograms of OrienTations).
A method of associating feature points that approximate feature amounts (luminance gradient vectors in the vicinity of feature points on an image, distribution of luminance gradient vectors of the entire image, and the like).
Which of these methods is adopted is determined by the map configuration. For example, when the map is composed only of the feature point position information, ICP is adopted if the map is two-dimensional, and SHOT is adopted if the map is three-dimensional. If the map is composed of feature point position information and feature quantities, the self-position estimation during the autonomous movement of the mobile 1 calculates the feature quantities using the same technique as that used when creating the map. Association using is performed.

The control unit (autonomous movement control unit) 116 determines the position and posture of the moving body 1 calculated by the self-position calculating unit 115, the corrected position of the target object, and the unit per unit time of the target object when the target object is a moving object. The next movement amount of the moving body 1 is determined based on the movement amount. Then, the control unit 116 calculates a control value for the driving unit 130 based on the next movement amount, and transmits the calculated control value to the driving unit 130.
The arrow A2 will be described later.

Next, with reference to FIG. 2A to FIG. 2C, errors that occur in the calculation of the monocular distance in the moving direction, the lateral direction, and the superstrate turning will be described.
(Rejection error by monocular distance calculation)
2A to 2C show details of distance calculation in the monocular distance calculation unit 113 and distance errors.
2A, it is assumed that the moving body 1 is moving in the direction of arrow A1 in FIG. 2A illustrates an error that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the front image captured by the image sensor 120A or the image sensor 120B.
In FIG. 2A, reference numeral 201 denotes the position and posture of the moving body 1 at time t1. At this time, the image sensor 120 A captures an image of the object J ahead of the moving direction of the moving body 1 with the line of sight 202.

Thereafter, the moving body 1 moves to a point 211 at time t 2, and the image sensor 120 A images the object J with the line of sight 212.
The monocular distance calculation unit 113 determines the distance to the object J based on the position and orientation at time t1 (reference numeral 201), the position and orientation at time t2 (reference numeral 211), and the parallax of the line of sight 202 and line of sight 212 at each time. (Line segment length of the line of sight 212) is calculated.
The parallax between the line of sight 202 and the line of sight 212 is calculated by detecting the same point included in the object J from the image at time t1 and the image at time t2. The same point is detected by using an optical flow estimation method. The position and orientation (reference numeral 211) at the time t2 are the control values for the drive unit 130 determined by the control section 116 at the time t1 to the position and orientation (reference numeral 201) at the time t1 calculated by the self-position calculator 115 described later. Is a provisional value of the position and orientation calculated by integrating. The provisional value of the position and orientation (reference numeral 211) at time t2 is processed and corrected by the self-position calculation unit 115. That is, the self-position calculation unit 115 corrects the position / posture of the moving body 1 at time t2 by using the map.

Here, it is assumed that the line of sight 212 at time t2 has captured the object J as the line of sight 221 due to the sensor error of the image sensor 120A. In this case, it is calculated that the object J exists at the point Ja. For this reason, an error E1 occurs in the distance to the object J.
A similar error occurs when a rear image captured by either one of the

image sensors

120C and 120D is used.

As described above, when the distance between the object J existing in the moving direction of the moving body 1 and the moving body 1 is obtained by monocular distance calculation, a large error occurs.

2B, it is assumed that the moving body 1 is moving in the direction of arrow A1 in FIG. 2B shows errors that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the side image captured by either the image sensor 120E or the image sensor 120F. ing.
The method for calculating the distance is the same as in FIG. 2A. That is, the object J is imaged at the line of sight 302 from the point 301 at time t1. Thereafter, the object J is imaged from the point 311 at the time t2 with the line of sight 312. Then, the distance to the object J (line segment length of the line of sight 312) is calculated. Here, if the line of sight 312 at time t2 images the object J as the line of sight 321 due to the sensor error of the image sensor 120E, it is calculated that the object J exists at the point Jb. That is, an error E2 occurs in the distance to the object J.

The error angle θ1 shown in FIG. 2A and the error angle θ2 shown in FIG. 2B are almost the same. Rather, the error angle θ2 shown in FIG. 2B is slightly larger than the error angle θ1 shown in FIG. 2A. Nevertheless, when the error E1 shown in FIG. 2A is compared with the error E2 shown in FIG. 2B, the error E2 shown in FIG. 2B is smaller. This is because the distance to the object J is inversely proportional to the parallax between the image at time t1 and the image at time t2. In short, if the parallax between the image at time t1 and the image at time t2 is small, the error tends to increase. Conversely, if the parallax between the image at time t1 and the image at time t2 is large, the error is difficult to increase.
Therefore, in order to reduce the error, it is necessary to increase the parallax between the two images as much as possible.

Therefore, it is desirable to apply the processing of the monocular distance calculation unit 113 having a lower processing load than the stereo distance calculation unit 112 to the imaging region in the direction orthogonal to the movement direction.
As described above, it is desirable that the monocular distance calculation unit 113 calculates the distance of the image captured by the image sensor 120 provided in a direction orthogonal to the moving direction of the moving body 1. For example, if the drive unit 130 is an omni wheel or the like, the moving body 1 can also move in the direction of the arrow A2 in FIG. When the moving body 1 moves in the direction of the arrow A2 in FIG. 1, the image sensors 120A to 120D are the image sensors 120 installed in a direction orthogonal to the moving direction. In such a case, distances calculated by the monocular distance calculation unit 113 may be performed on images captured by the image sensors 120A to 120D. However, in this case, an image captured by either one of the

image sensors

120A and 120B or one of the

image sensors

120C and 120D may be used.

Further, an image captured by the image sensor 120 installed in the positive direction (that is, forward) and the negative direction (that is, rearward) with respect to the moving direction of the moving body 1 is processed by the stereo distance calculation unit 112.
That is, since the error E1 is a large value as shown in FIG. 2A, the images captured by the

image sensors

120A and 120B installed in the moving direction of the moving body 1 as shown in FIG. From 113, the stereo distance calculation unit 112 is preferably processed.
As described above, the feature of the present application is that the monocular distance calculation or the stereo distance calculation is switched even in the same image sensor 120 depending on the moving direction of the moving body 1.

In FIG. 2C, the case where the mobile body 1 is performing a super turn is demonstrated.
When the mobile body 1 makes a super turn in the direction of the turning direction (arrow A3) in FIG. 2C, it can be said that the mobile body 1 is moving in a direction orthogonal to the movement direction indicated by the arrow A1 in FIG. In other words, in this case, the image sensors 120A to 120D are the image sensors 120 (image sensors 120 installed on the sides) that exist in a direction orthogonal to the moving direction of the moving body 1.

That is, it is better not to apply the processing of the stereo distance calculation unit 112 to the image sensors 120A to 120D while the mobile body 1 is turning around. That is, when the moving body 1 is turning in a super-confidence, the process of the monocular distance calculation unit 113 is applied to an image captured by at least one of the

image sensors

120A and 120B. Similarly, the processing of the monocular distance calculation unit 113 is applied to an image captured by at least one of the

image sensors

120C and 120D. The reason is described below.

In FIG. 2C, it is assumed that the moving body 1 is turning in the direction of arrow A3 in FIG. FIG. 2C shows an error that may occur when the monocular distance calculation unit 113 calculates the distance of the object J based on the image captured by the image sensor 120A or the image sensor 120B.

The distance calculation method in FIG. 2C is the same as in FIGS. 2A and 2B. At time t 1, the image sensor 120 A captures an image of the object J with the line of sight 402 in the position / posture 401. Thereafter, the mobile body 1 performs a super-spinning in the super-spinning direction (arrow A3). Then, at time t 2, the image sensor 120 A captures an image of the object J with the line of sight 412 at the position / posture 411.

Thereafter, the monocular distance calculation unit 113 calculates the distance to the target object J (line segment length of the line of sight 412) based on the image captured at time t1 and the image captured at time t2. It is determined from the control value of the control unit 116 at time t1 whether or not super turning is in progress.

Here, it is assumed that the visual line 412 at time t2 is captured as the visual line 421 by the sensor error of the image sensor 120A. In this case, it is calculated that the object J exists at the point Jc, and an error E3 occurs in the distance to the object J.

Here, when the error E1 shown in FIG. 2A is compared with the error E3 shown in FIG. 2C, the error E3 is smaller. This is because the amount of movement in the direction orthogonal to the movement direction, that is, the turning direction (arrow A3) increases, and the parallax increases. In other words, since the moving direction of the moving body 1 is the turning direction (arrow A3) in FIG. 2C, the image sensor 120A (or the image sensor 120B) is a side image sensor that captures the side of the moving body 1. This is because.

Therefore, the amount of super turning is calculated based on the control value of the control unit 116, and when the amount of super turning is larger than a preset determination threshold, the image is picked up by at least one of the

image sensors

120A and 120B. The processing of the monocular distance calculation unit 113 is applied to the image. In this case, the stereo distance calculation unit 112 does nothing. The same applies to the

image sensors

120C and 120D.

As described above, when the image sensor 120 is provided in a direction orthogonal to the moving direction of the moving body 1, the monocular distance calculation can make the error smaller than the stereo distance calculation. Furthermore, in the monocular distance calculation, since the stereo distance calculation can reduce the processing data, the processing load can be reduced.
That is, when the moving body 1 is turning in a super-confidence, a method of calculating the distance by the monocular distance calculation unit 113 rather than calculating the distance of the images captured by the

image sensors

120A and 120B by the stereo distance calculation unit 112. Can reduce the error. Furthermore, by calculating the distance with the monocular distance calculation unit 113, the amount of image data to be processed can be reduced, and the processing load can be reduced.

It should be noted that which of the

image sensors

120A and 120B the image captured by the image sensor 120 is applied to the monocular distance calculation unit 113 is determined based on the movement amount of each of the image sensors 120 due to the super turning. For example, of the

image sensors

120A and 120B, if the movement amount of the image sensor 120A is large, an image captured by the image sensor 120A is used. The same applies to the

image sensors

120C and 120D.
However, when the

image sensors

120A and 120B are in a symmetrical positional relationship with respect to the central axis of the moving body 1, the amount of movement of each of the

image sensors

120A and 120B due to the super turning is the same. In such a case, an image of the image sensor 120 having many feature points on the captured image is used. The same applies to the

image sensors

120C and 120D.

The determination threshold value for determining whether the monocular distance calculation unit 113 calculates the distance or the stereo distance calculation unit 112 calculates the distance is set in advance for each of the

image sensors

120A and 120B and the

image sensors

120C and 120D. For example, the distance (baseline distance) between the image sensor 120A and the image sensor 120B is set as the determination threshold value. The

image sensors

120C and 120D are set similarly. Here, the baseline distance is a parallax in the stereo distance calculation. That is, when images are captured at different times, when the distance that the moving body 1 moves in the direction of the arrow A2 in FIG. 1 is greater than the baseline distance between the respective imaging times, either of the

image sensors

120A and 120B is used. On the other hand, monocular distance calculation is applied to the captured image. When the distance that the moving body 1 has moved in the direction of the arrow A2 in FIG. 1 is smaller than the baseline distance between the respective imaging times, stereo distance calculation is performed on images captured by the image sensor 120A and the image sensor 120B. Applied. The same applies to the

image sensors

120C and 120D.

Incidentally, since the

image sensors

120E and 120F are provided in a direction orthogonal to the moving direction of the moving body 1 even during the super turning, distance calculation by the monocular distance calculation unit 113 is applied.

In addition, since CNN is an estimated distance by learning, calculation of monocular distance using CNN does not guarantee the accuracy of distance. In the present embodiment, since the distance is determined geometrically, an error occurs, but the accuracy of the distance calculated from the CNN is guaranteed.

It should be noted that although super-superficial turning is described here, it can also be applied to curves. When the moving body 1 is curved, when the moving direction vector of the moving body 1 is decomposed into a straight direction and a vector orthogonal to the straight direction (orthogonal direction), the straight direction component is larger than the orthogonal direction component. Stereo distance calculation is applied to an image captured by one of the

image sensors

120A and 120B. In addition, when the orthogonal direction component is larger than the straight traveling direction component, the monocular distance calculation is applied to the image captured by one of the

image sensors

120A and 120B. The same applies to the

image sensors

120C and 120D.

(Movement control)
FIG. 3 is a diagram illustrating a method of movement control in the moving body 1.
First, a target route W to the destination point is created based on the object J1 such as a wall and the destination location that are created in advance. Examples of the method for creating the target route W include the Dijkstra method and the A-Star method. In this method, first, a starting point and an ending point are designated. And the potential field P is provided in the vicinity of each feature point which comprises the target object J1. The potential field P is a region that the moving body 1 avoids. Next, the control unit 116 performs a route search with the minimum cost based on the distance to the destination point and the potential field P. Then, the control unit 116 generates a route having the minimum cost as the target route W. The target route W is transmitted to the mobile body 1 after being generated by, for example, a server (not shown).

Next, the control unit 116 sets a plurality of candidates B for the next moving speed and turning angular speed around the moving speed and turning angular speed B1 of the moving body 1 calculated based on the previous control value. And the control part 116 calculates the candidate position of the next mobile body 1 when each candidate B is employ | adopted.

Each moving speed and turning angular speed in each candidate B is set so as not to exceed the maximum moving speed and the maximum turning angular speed of the moving body 1. For example, it is assumed that the moving speed of the candidate B1 is Vm, and the moving speed of the candidate B2 is set in the range of (Vm−3 km / h) to (Vm + 3 km / h). At this time, if the maximum moving speed is (Vm + 2 km / h), the moving speed candidate B2 is reset to a range from (Vm−3 km / h) to (Vm + 2 km / h).

Next, based on the positions of the moving objects M1 and M2 corrected by the self-position calculating unit 115 and the movement amounts M11 and M21 per unit time, the control unit 116 generates a potential field P1 in the vicinity of the moving objects M1 and M2. , P2. Then, the control unit 116 compares the cost from the next movement destination candidate position of the mobile body 1 to the target route W with the cost from the next movement destination candidate position of the mobile body 1 to the target point (movement end point). At this time, the control unit 116 calculates the cost using a route that avoids the potential fields P, P1, and P2.
And the control part 116 selects the candidate B with the lowest cost. And the control part 116 employ | adopts the candidate B used as the minimum cost as a next moving speed and turning angular velocity. Thereafter, the control unit 116 calculates a control value for the drive unit 130 based on the adopted moving speed and turning angular velocity.

In addition, in the example of FIG. 3, in order to avoid the moving objects M1 and M2, the place where the moving body 1 has already deviated from the target route W is shown.

In the first embodiment, when the image sensor 120 is installed in the moving direction of the moving body 1, stereo distance calculation is applied. Further, when the image sensor 120 is installed in a direction orthogonal to the moving direction, monocular distance calculation is applied. By doing in this way, the mobile body 1 selects the optimal distance calculation method so that processing load may become low according to a movement / turning direction. Thereby, it is possible to realize autonomous movement with high accuracy and low processing load while having no blind spots. Further, by selecting a distance calculation method that reduces the error, it is possible to prevent the accuracy of the distance calculated even in an inexpensive camera from being lowered.
Thereby, it is possible to prevent the obstacle detection accuracy from being lowered even with an inexpensive camera.
Note that the technique described in Patent Document 1 monocularly views a portion where the field of view of the camera does not overlap regardless of the moving direction of the vehicle. On the other hand, the present embodiment is different in that the stereo distance calculation and the monocular distance calculation are switched according to the moving direction of the moving body 1.

In the past, the processing frequency was reduced by reducing the frequency of image acquisition. However, there is a possibility that performance degradation such as a decrease in distance calculation accuracy may occur in such a method. In the present embodiment, it is possible to prevent a decrease in distance calculation accuracy and the like without reducing the image acquisition frequency.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG.
In the second embodiment, the mobile body 1 starts and resumes autonomous movement safely at the time of start-up, stop after arrival at the destination, stagnation due to crowds, or temporary stop in elevator boarding, etc. With the goal.

FIG. 4 is a diagram illustrating a configuration of the calculation processing unit 110a of the moving body 1 according to the second embodiment.
The calculation processing unit 110a includes a state determination unit 117 and a turning angular velocity change unit (autonomous movement control unit) 118 in addition to the units 111 to 116 having the same configuration as that of the first embodiment.
Each unit 111 to 116 autonomously moves the moving body 1 by executing the same processing as that of each unit 111 to 116 in the first embodiment.
The state determination unit 117 determines the traveling state of the mobile body 1 based on the time series information of the position and orientation of the mobile body 1 calculated by the self-position calculation unit 115 and the activation information. When the calculation processing unit 110a is started up or when both the position and posture of the moving body 1 have not changed during a past fixed period (for example, 30 seconds) including the current time, the state determination unit 117 sets the running state as “ Determined as “stopped / stagnation” In other cases, the state determination unit 117 determines the traveling state as “traveling”.

The turning angular velocity changing unit 118 sets the maximum turning angular velocity of the moving body 1 based on the traveling state determined by the state determining unit 117. When the running state is “stop / stagnation”, the maximum turning angular velocity is set to zero. This is because the monocular distance cannot be calculated unless the images are acquired in time series. That is, when the running state is “stop / stagnation”, the distance of the object cannot be calculated by the

image sensors

120E and 120F. Thereby, since the distance of the object which exists in the side of the mobile body 1 cannot be calculated, the side of the mobile body 1 becomes a blind spot. Therefore, if the mobile body 1 is turned and started, there is a possibility of colliding with an unexpected obstacle. Therefore, the maximum turning angular velocity is set to zero and the vehicle can start only in the straight direction.

When the running state is “running”, the distance between the position of the moving body 1 and the current position of the moving body 1 at a time when the running state is switched from “stop / stagnation” to “running” is a fixed amount (for example, 1 m) Is exceeded, the maximum turning angular velocity is set to the original value of the moving body 1. This is because, as described above, for calculating the monocular distance, the images need to be aligned to some extent in time series.

The moving body 1 of the second embodiment controls the drive unit 130 so as not to turn until a certain amount of straight travels / retreats when starting or resuming running from the time of starting or stopping / stagnation. By doing in this way, until it becomes possible to accurately calculate the distance to the object existing on the side of the moving body 1, the object existing in front of the moving object 1 and the object existing behind the moving object 1 can be calculated. Move while calculating the distance by calculating the stereo distance. By doing in this way, the mobile body 1 can be moved in a safe direction.

The determination as to whether or not the position and orientation of the moving body 1 has been changed is performed, for example, by the following procedure. First, the state determination unit 117 calculates the variance between the position and the posture in a past fixed period including the current time. And the state determination part 117 performs the determination whether the position and attitude | position of the mobile body 1 are changing by comparing the state determination threshold value preset and the calculated dispersion | distribution.

Here, when the state determination threshold is set to be small, it is determined that the variance is stopped when the variance is close to zero. If the state determination threshold is set to be large, it is determined that the moving body 1 is stopped even when the moving body 1 is moving considerably.
If the state determination threshold is set to be small, it is determined that the moving body 1 is moving even if it moves a little. Therefore, before the image sensor 120 installed on the side of the moving body 1 moves a distance necessary for calculating the monocular distance, the monocular distance calculation by the image sensor 120 installed on the side is performed.
On the other hand, if the state determination threshold is set large, as described above, it is determined that the moving body 1 is stopped even when the moving body 1 is moving considerably. That is, after the image sensor 120 installed on the side of the moving body 1 moves a distance necessary for the monocular distance calculation, the monocular distance calculation is performed by the image sensor 120 installed on the side.

Because of these characteristics, it is better to set the state determination threshold to a large value when there are many people or obstacles around it, and to set the state determination threshold to a small value in places where there is nothing around. In this way, efficient movement of the moving body 1 can be realized.

[Third Embodiment]
FIG. 5 is a diagram illustrating a configuration example of the autonomous mobile system Z according to the third embodiment.
As shown in FIG. 5, the autonomous mobile system Z includes a moving body 1 b and a server 5.
The moving body 1b transmits an image captured by each image sensor 120 to the server 5. The server 5 calculates the distance from the moving body 1b to the object using monocular distance calculation or stereo distance calculation based on the transmitted image. Then, the server 5 calculates the position of the object based on the calculated distance to the object, and further calculates the self position (position / posture) of the moving body 1b by map matching. And the server 5 calculates the next moving speed and turning angle of the moving body 1b based on the self-position etc. of the moving body 1b. Thereafter, the server 5 transmits the self position of the moving body 1b and the next moving speed and turning angle of the moving body 1b to the moving body 1b as movement instruction information. The moving body 1b moves according to the received movement instruction information. The communication between the mobile body 1b and the server 5 is preferably wireless communication.
In addition to the processing described above, the server 5 also includes a stereo distance calculation unit 112, a monocular distance calculation unit 113, an object detection unit 114, a self-position calculation unit 115, and a first embodiment as described later in FIG. The processing performed by the control unit 116 is appropriately performed.

The calculation processing unit 110b of the moving body 1b includes a data transmission unit 119a, a data reception unit 119b, and a movement control unit 119c.
The data transmission unit 119 a transmits the image captured by each image sensor 120 to the server 5.
The data receiving unit 119 b receives movement instruction information from the server 5.
Further, the movement control unit 119 c controls each driving unit 130 based on the movement instruction information sent from the server 5.
Since the image sensor 120 (120A to 120F) and the drive unit 130 (130A to 130D) are the same as those in FIG. 1, description thereof is omitted here.
Details of the server 5 will be described with reference to FIG.

FIG. 6 is a diagram illustrating a configuration example of the server 5 according to the third embodiment.
The server 5 includes a data reception unit 511, a stereo distance calculation unit 512, a monocular distance calculation unit 513, an object detection unit 514, a self-position calculation unit 515, a control unit 516, and a data transmission unit 517.
The data reception unit 511 receives images captured by the respective image sensors 120 from the moving body 1b.
The stereo distance calculation unit 512, the monocular distance calculation unit 513, the object detection unit 514, and the self position calculation unit 515 are the stereo distance calculation unit 112, the monocular distance calculation unit 113, the object detection unit 114, and the self position calculation unit illustrated in FIG. Since the same operation as 115 is performed, a description thereof is omitted here.
The control unit 516 performs operations other than the control of the drive unit 130 among the operations performed by the control unit 216 of FIG.
The data transmission unit 517 includes a stereo distance calculation unit 512, a monocular distance calculation unit 513, an object detection unit 514, a self-position calculation unit 515, a self-position of the moving object 1b calculated by the processing by the control unit 516, and a moving object. The next moving speed / turning angle 1b is transmitted to the moving body 1b as movement instruction information.

The server 5 includes a CPU (not shown), a RAM (not shown), an HD (Hard Disk) (not shown), and the like. Then, the program stored in the HD or the like is loaded into the RAM, and the loaded program is executed by the CPU. By doing so, the data receiving unit 511, the stereo distance calculating unit 512, the monocular distance calculating unit 513, the object detecting unit 514, the self-position calculating unit 515, the control unit 516, and the data transmitting unit 517 are realized.

According to the third embodiment, since the calculation of the distance required for the movement of the moving body 1b and the position calculation of the moving body 1b are performed by the server 5, the processing load on the moving body 1b can be reduced. Incidentally, the server 5 can calculate the distance and the position of the moving body 1b for the plurality of moving bodies 1b.
Further, at least one of the stereo distance calculation unit 512, the monocular distance calculation unit 513, the object detection unit 514, the self-position calculation unit 515, and the control unit 516 configuring the server 5 illustrated in FIG. May be.

For example, an omni wheel can be used as the drive unit 130. In such a case, the moving body 1 can move in the direction of the image sensor 120E or the image sensor 120F in FIG. In such a configuration, two image sensors 120 may be provided in the direction of the image sensor 120E or the image sensor 120F.

The moving body 1 of the present embodiment is applicable to autonomous mobile devices such as robots, vehicles that move autonomously, personal mobility that moves autonomously, and the like.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Further, each of the above-described configurations, functions, units 111 to 118, 511 to 517, a storage unit, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Further, each configuration, function, and the like described above may be realized by software by a processor such as a CPU interpreting and executing a program that realizes each function. Information such as programs, tables, and files for realizing each function is stored in an HD (Hard Disk), a memory, a recording device such as an SSD (Solid State Drive), an IC (Integrated Circuit) card, It can be stored in a recording medium such as an SD (Secure Digital) card or a DVD (Digital Versatile Disc).
In each embodiment, control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it can be considered that almost all configurations are connected to each other.

1 Mobile body (autonomous mobile device)
5

servers

112, 512 Stereo distance calculation unit (distance calculation unit)
113,513 Monocular distance calculation unit (distance calculation unit)
114,514 Object detection unit (autonomous movement control unit)
115,515 Self-position calculation unit (autonomous movement control unit)
116,516 Control unit (autonomous movement control unit)
117, State determination unit 118 Turning angular velocity change unit (autonomous movement control unit)
120, 120A to 120F Image sensor (imaging unit)

Claims

Having at least two first imaging units including, in an imaging region, a direction in the positive direction with respect to the straight traveling direction of the moving body;
Each having at least one second imaging unit including an orientation in the imaging region that is orthogonal to the straight direction of the moving body;
A distance calculation unit that calculates a distance to an object in the imaging region based on a captured image captured by the first imaging unit and the second imaging unit;
An autonomous movement control unit that calculates a self-position based on the calculated distance, and performs autonomous movement based on the calculated self-position;
The distance calculation unit
Calculating the distance based on the parallax of the image captured by the first imaging unit;
The autonomous mobile device, wherein the distance is calculated based on a plurality of captured images captured by the second imaging unit and having different imaging times.
A state determination unit for determining whether the autonomous mobile device is stopped or moving;
The autonomous movement control unit
2. The autonomous mobile device according to claim 1, wherein when the autonomous mobile device moves from stop to move, the autonomous mobile device is moved in a straight direction for a predetermined distance.
The determination of whether the autonomous mobile device is stopped or moving is performed by comparing the dispersion of the position of the autonomous mobile device with a determination threshold,
The autonomous mobile device according to claim 2, wherein the determination threshold is changed according to a surrounding environment in which the autonomous mobile device moves.
The first imaging unit includes at least two third imaging units including, in an imaging region, a negative direction with respect to the straight direction of the moving object.
The distance calculation unit
The autonomous mobile device according to claim 1, wherein the distance is calculated based on parallax of an image captured by the third imaging unit.
An autonomous mobile device and a server,
The autonomous mobile device is
Having at least two first imaging units including, in an imaging region, a direction in the positive direction with respect to the straight traveling direction of the moving body;
Each has at least one second imaging unit including an orientation in the imaging region that is orthogonal to the straight direction of the moving body,
Based on the autonomous position of the autonomous mobile device sent from the server, an autonomous movement control unit that performs autonomous movement,
The server
A distance calculation unit that calculates a distance to an object in the imaging region based on a captured image captured by the first imaging unit and the second imaging unit;
A position calculation unit that calculates a self-position of the autonomous mobile device based on the calculated distance;
Have
The distance calculation unit
Calculating the distance based on the parallax of the image captured by the first imaging unit;
The autonomous movement system, wherein the distance is calculated based on a plurality of captured images captured by the second imaging unit and having different imaging times.