JP7461737B2 - Mobile object and its navigation system - Google Patents

Description

The present invention relates to an autonomously moving vehicle and its navigation system, and in particular to a system in which a learning device learns approach movement command formulas and alignment command formulas using data collected by a data-collecting vehicle, and generates commands for the vehicle to approach and align to a target position using the learned approach movement command formulas and alignment command formulas.

In recent years, mobile objects that move autonomously while transporting goods, providing guidance, and carrying out tasks have begun to operate in many indoor facilities, such as factories, hospitals, and public facilities. Navigation for such mobile objects can be achieved in a variety of ways, including installing a guide system (indoor GPS, markings, etc.) in the environment, or using a map created from environmental information (shape, images, etc.) collected in advance by moving around the driving environment, and moving to a target location while matching the currently acquired environmental information with landmark information on the map. However, advance preparations such as the maintenance of the environment, the creation of a highly reliable map, and software development work for map matching, etc., are a heavy burden.

As a navigation technology that solves this problem, for example, claim 1 of Patent Document 1 discloses a method for controlling a moving body, which includes comparing a teaching image captured in advance on a travel path of a moving body equipped with an imaging device with an actual image captured by the imaging device, and controlling the travel of the moving body so that the degree of match between the actual image and the teaching image increases, the method comprising: a steady travel step for causing the moving body to travel at a steady speed based on the teaching image and the actual image; a low-speed travel step for causing the moving body to travel at a deceleration speed lower than the steady speed when the degree of match between a deceleration start position image preset from among the plurality of teaching images and the actual image reaches a preset first threshold; a positioning step for causing the moving body to travel at a positioning speed lower than the deceleration speed when the degree of match between a positioning start position image preset from among the plurality of teaching images and the actual image reaches a preset second threshold; and a stop step for stopping the drive unit when the degree of match between a target position image preset from among the plurality of teaching images and the actual image reaches a preset third threshold. According to the navigation technology disclosed in Patent Document 1, images are acquired at multiple positions up to the target stopping position as a preliminary step, and in particular, by simply preparing multiple images at positions where the vehicle will decelerate in the direction of travel (forward/backward), the vehicle can decelerate in multiple stages, improving the accuracy of the stopping position.

JP 2010-140080 A

However, the navigation technology of Patent Document 1 controls the left and right position of a moving object using the degree of agreement between an image captured in advance (target position image/instruction image) and an image captured during driving (actual image) and the difference between the positions of feature points in the instruction image and the positions of corresponding feature points in the actual image. In this respect, the technology can move to a target position with high accuracy in an environment with relatively small environmental changes, such as a factory corridor (the number of objects, changes in their placement, the presence or absence of passersby, the brightness of lighting, etc.) and in linear movement near the target stopping position. However, when applied to an environment with large environmental changes such as a shopping mall or public facility, or a wide moving environment, or when applied to a highly accurate moving method for the position and attitude (direction, orientation) of a moving object in a narrow environment, or when applied to a different type of autonomous moving object such as a moving object that can move in all directions (holonomic) or a moving object that cannot move sideways like a car (nonholonomic), problems such as a decrease in movement accuracy, processing failure, and an increase in the load of advance preparation such as parameter tuning may arise, and it may not be possible to stop accurately at the target stopping position.

As a more specific example, the technology described in Patent Document 1 controls a moving object by calculating the degree of match between images and comparing feature points between images. Therefore, environmental changes (such as the placement of people and objects, lighting conditions, etc.) between when the teaching image is acquired and when the actual image is acquired can cause changes in the degree of match and changes in the characteristics of feature points or their disappearance (due to lighting, shadows, occlusion, etc.), resulting in problems such as reduced movement accuracy and control failure in dynamic environments such as shopping malls and public facilities. This problem is particularly evident in situations where there is a distance between the moving object and surrounding objects, or when moving over a wide area with a distance to the target position, there is little change between the teaching image and the actual image relative to the amount of movement of the moving object, and people and other objects are easily captured between the moving object and surrounding objects.

At the same time, the technology described in Patent Document 1 sequentially controls the speed and lateral position of a moving object by calculating the degree of coincidence of images and comparing feature points between images, and therefore cannot handle cases where precision in orientation as well as position is required, such as when passing through a narrow space such as a door (passing through an open door), and where planning of position and orientation (position and orientation) on the path to a target position is required (such as planning for parking a car in a garage), and there is a problem in that high positional precision cannot be achieved. This problem is particularly noticeable with nonholonomic moving objects, and there is a problem in that it cannot handle various moving methods and moving objects.

At the same time, the technology described in Patent Document 1 requires that a speed plan (travel plan) be made in advance and then teaching images be acquired at the deceleration position (control position), which poses the problem that the burden of advance preparation increases every time the travel method or moving body is changed.

The present invention has been made in light of this background, and aims to provide a navigation device for a moving body that requires little advance preparation, does not require facility construction or map creation, and achieves robust movement in dynamic and wide moving environments, and highly accurate movement in narrow spaces where path planning of position and attitude is required, and requires little advance preparation for various types of movement methods and autonomous moving bodies.

In order to solve the above-mentioned problems, the moving body of the present invention is a moving body that includes an image acquisition unit that captures a current position image, a target position image storage unit that stores a target position image captured at a target position, an approach movement command unit that generates a movement command to approach and move to the target position based on an approach movement command formula, an alignment command unit that generates a movement command to stop at the target position or a movement command to pass through the target position based on an alignment command formula, a movement switching unit that selects either the approach movement command unit or the alignment command unit depending on the output result of the approach movement command formula or the alignment command formula, and a movement control unit that controls a movement mechanism based on the movement command , wherein the approach movement command formula is composed of an equation that outputs an object position and object type in an image when an image is input, and the alignment command formula is composed of an equation that outputs a relative position and relative attitude of the imaging positions when images captured at two positions are input .

The present invention provides a navigation device that requires little advance preparation, as it does not require facility construction or map creation, and that realizes robust movement in dynamic and wide moving environments, and highly accurate movement in narrow spaces where path planning of position and attitude is required, and can provide a navigation device for moving bodies that requires little advance preparation for various types of moving methods and autonomous moving bodies.

FIG. 1 is a diagram showing an example of the configuration of a moving body in a navigation system according to an embodiment; FIG. 2 is a diagram showing a configuration example of a data collecting vehicle in the navigation system according to the embodiment; FIG. 2 is a diagram showing an example of the configuration of a learning device of the navigation system according to an embodiment; 4 is a flowchart showing the overall process of the navigation system according to the embodiment. FIG. 13 is a diagram illustrating an example of a collection range of images used to generate learning data for approach movement. FIG. 13 is a diagram illustrating an example of a collection range of learning data for alignment. 4 is a flowchart showing a learning procedure of the learning device according to the embodiment. 4 is a flowchart showing a travel procedure of a moving body according to an embodiment. FIG. 1 is a diagram showing an example of a navigation device in which a plurality of moving objects 1 exist.

The following describes an embodiment of the present invention with reference to the drawings as appropriate.

The navigation system 100 of the present invention is a system composed of a mobile body 1 that moves autonomously toward a target position, a data-collecting mobile body 2 that collects learning data, and a learning device 3 that learns an approach movement command formula and an alignment command formula using the collected learning data. The mobile body 1 generates movement commands for autonomous movement using the approach movement command formula and the alignment command formula learned by the learning device 3, and moves autonomously to the final target position while passing through specified waypoints. The learning device 3 can share information with the mobile body 1 and the data-collecting mobile body 2 via wired or wireless communication.

In the following, it is assumed that the moving body 1 is either a nonholonomic moving body (such as a vehicle or a cart) that cannot move sideways, or a holonomic moving body (such as a robot or a drone) that can move in all directions, and that the environment in which the moving body 1 moves autonomously is an environment with large changes in the amount of human traffic and brightness, such as a shopping mall or a public facility. Even in such a situation, the present system can generate the movement commands required for autonomous movement (autonomous driving, autonomous walking, autonomous flying, etc.) to the target position using a common approach movement command formula and positioning command formula regardless of the form of the moving body 1, so that the learning device 3 has the advantage that it is not necessary to learn a command formula for each form of the moving body 1. Below, the embodiments of the moving body 1, the data collection moving body 2, and the learning device 3 in the navigation system 100 of the present invention will be described in detail with reference to the drawings.

<Mobile object 1>
1 is a diagram mainly for explaining an example of the configuration of a mobile body 1, and also shows some configurations of a data-collecting mobile body 2 and a learning device 3. Note that, although a navigation system 100 having only one mobile body 1 is illustrated here, the system may have a plurality of mobile bodies 1 of different configurations, as will be described later with reference to FIG.

As shown in FIG. 1, the moving body 1 has an image acquisition unit 11, a moving mechanism 12, a calculation unit 13, and a memory unit 14. Each of these will be described in detail in turn.

The image acquisition unit 11 acquires images of the surroundings from the current position of the moving body 1, and is composed of one or more cameras equipped with imaging elements such as a CCD image sensor or a CMOS image sensor. The images of the outside world captured by the image acquisition unit 11 are output to the movement switching unit 13a, global movement command unit 13b, approach movement command unit 13c, and alignment command unit 13d, which will be described later.

The image acquisition unit 11 desirably uses a high-resolution camera, and may be equipped with a camera equipped with a wide-angle lens to capture a wide range of the moving environment. The viewing angle and resolution of the camera desirably have the same conditions as those of the camera of the image acquisition unit 21 mounted on the data collection moving body 2 described below, but the conditions may be made the same by performing trimming and resizing processes. Furthermore, the camera is desirably attached at a position and in a direction that allows it to capture an image of the area ahead in the moving body's direction of travel, and is desirably fixed to the moving body 1 so that its attitude does not move, but if a system that constantly monitors the camera's attitude is installed, the camera's attitude does not need to be fixed.

The moving mechanism 12 is a mechanism that realizes the movement of the moving body 1, and is an actuator that drives the wheels if the moving body 1 is a vehicle, the legs or wheels if the moving body 1 is a robot, the propellers if the moving body 1 is a drone, and the wheels. The moving control unit 13e, which will be described later, generates a desired movement command (e.g., translational movement speed, rotational movement speed, etc.) according to the form of the moving mechanism 12 based on a movement control signal output from the alignment command unit 13d, which will be described later, etc., and moves the moving body 1 forward/backward, turns, or stops.

The calculation unit 13 has a movement switching unit 13a, a global movement command unit 13b, an approach movement command unit 13c, an alignment command unit 13d, and a movement control unit 13e. A predetermined program is stored in the memory unit 14 of the moving body 1, and the movement switching unit 13a, the global movement command unit 13b, the approach movement command unit 13c, the alignment command unit 13d, and the movement control unit 13e are realized by loading this program into memory and executing it by the CPU (Central Processing Unit).

The movement switching unit 13a compares the current position image acquired by the image acquisition unit 11 with the target position image acquired from the target position image storage unit 14a, and judges whether to switch the movement method to the target position to one of "global movement" suitable for high-speed movement, "approach movement" for approaching and moving in the direction of the target position, or "alignment" for stopping accurately at the target position or for passing through the target position accurately, and outputs the judgment result to the global movement command unit 13b, the approach movement command unit 13c, and the alignment command unit 13d. Note that, although an example is given here of three types of movement methods, the present invention does not limit the number of movement methods. An example of the judgment of switching between each movement method by the movement switching unit 13a is explained below.

When switching the movement method to "approach movement", the movement switching unit 13a, for example, judges whether a pre-specified object (landmark) captured in the target position image is captured in the current position image, and selects switching to "approach movement" when it is judged that the object is captured. Note that the role may be divided such that the approach movement command unit 13c performs a judgment based on the image, and the movement switching unit 13a judges the switching based on the judgment result of the approach movement command unit 13c. At this time, the target position image used by the approach movement command unit 13c is switched near the waypoint, and while switching the target position image, the movement moves to the point where the final target position image was captured. Furthermore, if an object in the current position image is captured in multiple target position images, the target position image closest to the final target position is selected.

When switching the movement method to "alignment", the movement switching unit 13a, for example, determines whether the similarity between the current position image and the target position image for alignment is equal to or greater than a threshold, and selects switching to "alignment" when the similarity is equal to or greater than the threshold. The method of calculating the similarity of the images is, for example, MSE (Mean Square Error), which tallies the squares of the differences in luminance of pixels at the same positions in two images. The output of the alignment command unit 13d may be used as part of the processing of the similarity. If the same object is captured in the target position image used by the approach movement command unit 13c and the target position image used by the alignment command unit 13d, the movement method may be switched to "alignment" if the size of the object output by the approach movement command unit 13c is equal to or greater than a preset threshold, or the movement method may be switched to "alignment" if the relative position and orientation output of the alignment command unit 13d is continuously outputting values equal to or less than the threshold.

If none of the above applies, the movement switching unit 13a selects "global movement" as the movement method. When multiple movement methods overlap and it is determined that they are compatible, the order of priority is "alignment," "approach movement," and "global movement." Each determination may be performed simultaneously in parallel, or the order of determination may be determined. In this way, in this embodiment, an appropriate method can be selected from multiple movement methods, and it is possible to switch to another movement method at an appropriate time.

The global movement command unit 13b is a command unit that is selected when the movement switching unit 13a judges that "global movement" is to be performed. It receives the current position image acquired from the image acquisition unit 11 as an input, and outputs movement commands such as speed commands for moving in the direction of the target position to the movement control unit 13e. The movement method by the global movement command unit 13b is not particularly limited in this embodiment, but the moving object may move to the target position using a method called moving visual odometry that estimates the amount of movement of the moving object from changes in the current position image. This method is not highly accurate, but rough movement information can be obtained only from the time series input of the current position image. Alternatively, for example, a method called visual SLAM may be used. This method creates a map in which a group of feature points of the image is arranged three-dimensionally from the time series data of the current position image acquired by the image acquisition unit 11, while simultaneously estimating the current position of the moving object. This method cannot be used all the time because it requires a very large calculation load and accumulates errors.

The approach movement command unit 13c is a command unit that is selected when the movement switching unit 13a determines that an "approach movement" is to be performed, and inputs the current position image acquired by the image acquisition unit 11 and the target position image acquired from the target position image storage unit 14a into an approach movement command formula described below, and outputs the direction of the target position. Note that this approach movement command formula uses one obtained by prior learning in the learning device 3.

The alignment command unit 13d is a command unit that is selected when the movement switching unit 13a determines that "alignment" is to be performed, and inputs the current position image acquired by the image acquisition unit 11 and the target position image acquired from the target position image storage unit 14a into an alignment command formula described below, and outputs the relative position and orientation (relative position and relative angle) to the target position and orientation (including the orientation of the moving body). This alignment command formula uses one obtained by prior learning in the learning device 3.

The movement control unit 13e receives as input the movement command calculated by the global movement command unit 13b, the direction of the target position calculated by the approach movement command unit 13c, or the relative position and attitude to the target position and attitude calculated by the alignment command unit 13d, and performs a movement plan (or route plan) to realize the input command, etc., and performs appropriate movement control. By performing a movement plan from the relative position and attitude in the movement control unit 13e and performing movement control to the target position and attitude, it is possible to use a common command formula to handle any moving body 1 with a different type of moving mechanism 12, such as a holonomic moving body that can move in all directions, such as a drone, or a nonholonomic moving body that cannot move straight sideways, such as a car, even in a situation where high positional accuracy and high directional accuracy are required, such as when passing through a narrow space such as a door (opening a door and passing through).

The memory unit 14 has a target position image memory section 14a, an approach movement command type memory section 14b, and an alignment command type memory section 14c.

The target position image storage unit 14a stores one or more target position images captured in advance. In this embodiment, the moving body 1 is navigated to a position where the target position image stored in the target position image storage unit 14a was captured as a waypoint or a final target position. The target position image stored here may be captured by moving the moving body 1 or another moving body to be navigated in advance, or may be a target position image captured in advance by a person or a target position image on the Web or cloud. However, if the image acquisition unit 11 of the moving body 1 is composed of multiple cameras, it is necessary to acquire target position images with the same positional relationship. In this embodiment, it is preferable to use an image captured from the same height as the image acquisition unit 11, but it can also be used in cases where the installation position in the vertical direction is different, as described later. As a result, this embodiment can be applied to moving bodies with different installation positions of the image acquisition unit 11. In addition, the target position image storage unit 14a may store designation information for the target position image used by the positioning command unit 13d and the target position image used by the approach movement command unit 13c, and may store object information captured in the target position image used by the approach movement command unit 13c.

The approach movement command formula storage unit 14b appropriately acquires the "approach movement command formula" learned by the learning device 3 from the approach movement command formula storage unit 33c of the learning device 3. Similarly, the alignment command formula storage unit 14c appropriately acquires the "alignment command formula" learned by the learning device 3 from the alignment command formula storage unit 33d of the learning device 3. Note that the approach movement command formula and the alignment command formula are acquired, for example, when the learning device 3 performs re-learning.

Next, configuration examples of the data collection vehicle 2 and the learning device 3 will be described with reference to Figures 2 and 3.

<Data collection vehicle 2>
2 is a diagram mainly for explaining an example of the configuration of the data collection mobile body 2, and also shows some configurations of the mobile body 1 and the learning device 3. The data collection mobile body 2 is a device that collects learning data in the vicinity of a target position, and as shown in Fig. 2, has an image acquisition unit 21, a coordinate estimation sensor 22, a calculation unit 23, and a storage unit 24. Note that in this embodiment, since it is assumed that a system operator or the like collects learning data while moving the data collection mobile body 2, a moving mechanism is omitted in the data collection mobile body 2 in Fig. 2, but if it is desired to make the data collection mobile body 2 self-propelled, a moving mechanism may be provided as in the mobile body 1.

The image acquisition unit 21 acquires an image of the surroundings from the current position of the data collection mobile body 2, and is composed of one or more cameras equipped with an image sensor such as a CCD image sensor or a CMOS image sensor. The image data acquired by capturing the outside world by the image acquisition unit 21 is output to the positioning learning data creation unit 23a and the image storage unit 24a, which will be described later. In order to capture a wide range of the moving environment and collect a large amount of learning data, it is preferable to mount multiple cameras with different installation positions (positions relative to height and moving direction) and types, or to mount a camera equipped with a wide-angle lens. However, it is preferable to mount one of the multiple cameras at a position and direction that allows the camera to capture the image ahead in the traveling direction. It is preferable that the viewing angle and resolution of the camera are as large as possible. Furthermore, it is preferable to have a mechanism that can change the height and mounting angle (pitch angle and roll angle) of the camera. Furthermore, it is preferable to fix the camera so that its attitude does not move when it is mounted. However, if a system that constantly grasps the attitude of the camera is mounted, the attitude of the camera does not need to be fixed. In this way, one or a small number of data-collecting mobile objects 2 are used to collect various learning data, and this embodiment is characterized by the fact that learning results can be obtained that can respond to various conditions for different types of mobile objects.

The coordinate estimation sensor 22 acquires data necessary to estimate the coordinates of the data collection mobile body 2. For example, by using a sensor that acquires the number of wheel rotations, such as an encoder, the coordinates of the data collection mobile body 2 can be estimated from the cumulative number of wheel rotations. The coordinates of the data collection mobile body 2 may also be estimated using information from external sensors, such as a motion capture system or a beacon that generates radio waves.

The calculation unit 23 has a positioning learning data creation unit 23a and a coordinate estimation unit 23b. A predetermined program is stored in the memory unit 24 of the data collection vehicle 2, and the positioning learning data creation unit 23a and the coordinate estimation unit 23b are realized by loading this program into the memory and executing it by the CPU.

The coordinate estimation unit 23b estimates and outputs the coordinates and orientation of the position of the data collection mobile body 2 using the sensor data input from the coordinate estimation sensor 22. The coordinates (and orientation) here may be absolute coordinates based on a certain point, or may be relative coordinates within a series of alignment learning data.

The registration learning data creation unit 23a combines the image acquired by the image acquisition unit 21 with the coordinates (and orientation) input from the coordinate estimation unit 23b, and outputs the combined data as registration learning data. The registration learning data is a data set for learning the relationship between two images (input of the equation) and the amount of movement of the points at which each image was captured (relative position and orientation: output of the equation) to the registration command equation, and is composed of data that includes a set of images captured at multiple positions and the coordinates of the positions at which each image was captured. The registration learning data creation unit 23a associates the input image with the coordinates of the positions at which the image was captured as a set, and outputs the set to the registration learning data storage unit 24b.

The memory unit 24 has an image memory section 24a and a positioning learning data memory section 24b.

The image storage unit 24a temporarily stores images used to create approach movement learning data for learning the approach movement command formula described below. The images temporarily stored in the image storage unit 24a are transferred to the image storage unit 33a of the learning device 3 periodically or when learning is performed.

The alignment learning data storage unit 24b temporarily stores the alignment learning data input from the alignment learning data creation unit 23a. The alignment learning data temporarily stored in the alignment learning data storage unit 22b is transferred to the alignment learning data storage unit 33b of the learning device 3 periodically or when learning is performed.

<Learning device 3>
3 is a diagram mainly for explaining an example of the configuration of the learning device 3, and also shows some configurations of the moving body 1 and the data collecting moving body 2. The learning device 3 is a device that learns an approach movement command formula and an alignment command formula based on learning data transferred from the data collecting moving body 2, and as shown in Fig. 3, has an object region input section 31, a calculation unit 32 (approach movement learning data creation section 32a, approach movement learning preprocessing section 32b, approach movement command formula learning section 32c, alignment learning preprocessing section 32d, alignment command formula learning section 32e), and a memory unit 33 (image memory section 33a, alignment learning data memory section 33b, approach movement command formula memory section 33c, alignment command formula memory section 33d).

<Learning process of alignment command formula>
First, the alignment command formula learning process performed by the alignment learning data storage unit 33b, the alignment learning preprocessing unit 32d, the alignment command formula learning unit 32e, and the alignment command formula storage unit 33d will be described.

The positioning learning data storage unit 33b appropriately stores the positioning learning data transferred from the positioning learning data storage unit 24b of the data-collecting mobile unit 2. In addition, although not shown, if learning data already acquired by another device or on another occasion is available on the Web or cloud, positioning learning data is also stored from there.

Next, the alignment learning pre-processing unit 32d increases the variation of the alignment learning data in order to improve the estimation accuracy of the alignment command formula. For example, the variation of the learning data is increased by performing image processing related to lighting conditions such as contrast adjustment and gamma conversion, and robustness-related processing such as adding noise and deleting parts of the image on each image of the alignment learning data. Note that even if an image is transformed, the corresponding coordinates do not change. In this way, a feature of this embodiment is that when moving through a wide range of environments, learning results can be obtained that are robust to changes in the environment (position of people and objects, lighting conditions, etc.) between when the learning data images are acquired and when the actual images are acquired.

Here, the alignment command formula will be described in detail. The alignment command formula is a formula that takes images P and Q captured at positions p and q as inputs, and outputs the coordinates of position q when position p is the origin, that is, the amount of movement. First, there is (Formula 1) as an example of the alignment command formula. Note that, for the sake of simplicity, the explanation will be given using the simple (Formula 1), but (Formula 1) represents one layer of a deep neural network, and in reality, a more complicated calculation formula is used in which the output u ₁ (hereinafter, the subscript is the layer number) is further input to (Formula 1) as input x ₂ , and the output u ₂ is further input to (Formula 1) as input x ₃ , and this is repeated for the number of layers.

In (Equation 1), x is a vector representing images P and Q (hereafter referred to as two-point image data). f is a nonlinear function, u is a vector representing the amount of movement from position p to q, and is three-dimensional (X, Y, θ). u is a Cartesian coordinate system in which, when position p is the origin, X is the direction toward the front of the image, Y is the direction to the left of the image, and θ is positive with the front of the image being 0. And W is the weighting coefficient matrix for x. And b is a coefficient representing the amount of shift.

The learning process of the alignment command formula in this embodiment is a process of adjusting the coefficient matrix W and the shift amount b so that the movement amount u between two points output when two-point image data x is input outputs a more accurate determination result. As a result of learning, the moving body 1 becomes able to estimate the movement amount between two points from the two-point image data x by the alignment command formula using the learned coefficient matrix W and shift amount b in (Equation 1).

Now, the learning process of the alignment command formula performed by the alignment command formula learning unit 32e will be explained. First, some learning data are randomly extracted from the group of learning data for alignment input from the alignment learning pre-processing unit 32d. Then, the two-point image data x of each extracted data is input to (Equation 1) into which the initial value of the coefficient matrix W and the initial value of the shift amount b are substituted, and the error e between u calculated by (Equation 1) and the actual movement amount u' of the two points is calculated, and the total error value E is calculated. Then, the coefficient matrix W and the shift amount b are slightly changed so that the total error value E becomes smaller. For example, the gradient descent method can be used as a method for slightly changing the coefficient matrix W and the shift amount b here. This learning process is performed for all of the learning data group for alignment.

By repeating the above procedure, the total error value E gradually decreases, and the alignment command formula continues to be updated until the learning end condition is met. The learning end condition may be, for example, a condition in which the learning process for the group of alignment learning data is performed a certain number of times, or a condition in which the total error value E is the smallest. The amount of movement between the two points output by the alignment command formula thus obtained is considered to be the optimal amount of movement for the two-point image x. Therefore, by inputting the current position image and the target position image of the moving object 1 as x, the distance and direction to the target position can be calculated. Furthermore, by performing the learning process using a diverse and large amount of alignment learning data, it becomes possible to calculate the movement distance to the target position even in unfamiliar locations not included in the alignment learning data, or when there are environmental changes.

The alignment command formula (Formula 1) calculated in this manner, including the coefficient matrix W and shift amount b after the learning process is completed, is stored in the alignment command formula storage unit 33d. Then, at a timing such as before the operation of the mobile body 1, the alignment command formula is transferred to the alignment command formula storage unit 14c of the mobile body 1.

<Learning process of approach movement command type>
Next, the learning process of the approach movement command formula by the object area input unit 31, the image memory unit 33a, the approach movement learning data creation unit 32a, the approach movement learning pre-processing unit 32b, the approach movement command formula learning unit 32c, and the approach movement command formula memory unit 33c will be described.

The image storage unit 33a appropriately stores images transferred from the image storage unit 24a of the data collection vehicle 2.

Next, the approach movement learning data generating unit 32a creates approach movement learning data from the images stored in the image storage unit 33a. Specifically, the approach movement learning data creating unit 32a determines the range of the image input from the image storage unit 33a in which an object (e.g., a door) that can be a landmark for movement is captured, and specifies the object area portion in which the landmark is captured. The object area portion may be specified by a human through the object area input unit 31. In this case, the object area portion may be specified, for example, by using a so-called bounding box (a rectangle surrounding an object). The object area input unit 31 may use, for example, a mouse, a keyboard, a touch screen, or the like. Note that the object area portion may not be specified by a human, but may be automatically extracted from color, shape, or the like. By using the object area, it is possible to obtain size information of the object in addition to the position of the object. This set of object area information and the image becomes the approach movement learning data. At this time, although not shown, if learning data already acquired by another device or on another occasion is on the Web or cloud, the learning data obtained from there may be used.

Next, the approach movement learning pre-processing unit 32b increases the variation of the approach movement learning data in order to improve the estimation accuracy of the approach movement command formula. For example, the variation of the data is increased by performing image processing related to lighting conditions such as contrast adjustment and gamma conversion, and robustness-related processing such as adding noise and deleting part of the image on each image of the approach movement learning data. Note that even if an image is transformed, the corresponding object area is not changed. In this way, even for images that are not included in the learning data, a learning result that is robust to environmental changes (positioning of people and objects, lighting conditions, etc.) between the time the learning data image is acquired and the time the actual image is acquired is a feature of this embodiment.

Here, we will explain the approach movement command formula in detail. The approach movement command formula is composed of a two-stage formula. First, landmark objects are detected from the current position image and the target position image input into the approach movement command formula. Then, the position of the detected objects is used to calculate the direction of the target position.

The detection of an object, which is the first stage of the approach movement command formula, can be expressed, for example, by (Formula 2) in the same way as the alignment command formula (Formula 1). Note that, for the sake of simplicity of explanation, the simple (Formula 2) is used here as in the above, but (Formula 2) represents one layer of a deep neural network, and in reality, a more complicated calculation formula is used in which the output u' ₁ (hereinafter, the subscript is the layer number) is further input to (Formula 2) as input x' ₂ , and the output u' ₂ is further input to (Formula 2) as x' ₃ , and this repeated processing is performed the number of times equal to the number of layers.

In this case, x' is a vector of one image, and u' is a vector that represents the type of detected object, its certainty, and its position and size within the image. The certainty of a detected object is expressed as a number between 0 and 1 for each object detected; the larger the number, the more likely it is that object. The position and size of a detected object are expressed, for example, by the width, height, and pixel position of the center point of the bounding box.

Then, similar to the learning process for the alignment command formula, by learning (Equation 2) using the approach movement learning data, it becomes possible to detect landmark objects from within the image. At this time, it is also possible to simultaneously detect multiple objects by cutting the input image into multiple images. Fixed objects such as plants, signs, doors, lights, poles, and windows can be used as landmark objects, but it is better to use objects installed above to avoid the influence of moving objects such as people.

The second stage of the approach movement command formula can be expressed, for example, as follows (Formula 3).

In (Equation 3), u ₁ and u ₂ are pixel positions of objects (same object type) commonly detected from the current position image and the target position image. When detecting multiple objects, it is necessary to prepare an equation corresponding to the number of objects. Here, if detection in the height direction is not required, only the pixel position in the width direction of the image may be used. In this case, images captured at positions with different heights can also be used, and the imaging method of the target position image and the type of moving object can be freely selected. y is the direction of the target position, k is a coefficient matrix, and g is a function that calculates a geometric transformation.

The approach movement command formulas (Formula 2) and (Formula 3) calculated in this manner, including the coefficient matrix W' and shift amount b' after the learning process is completed, are stored in the approach movement command formula storage unit 33c. Then, at a timing such as before the operation of the mobile body 1, the approach movement command formulas are transferred to the approach movement command formula storage unit 14b of the mobile body 1.

<Flowchart of the navigation system 100>
Fig. 4 is a flowchart showing an example of a series of processes performed by the navigation system 100 of this embodiment, which is composed of the mobile object 1, the data collection mobile object 2, and the learning device 3. Note that Fig. 4 shows an example for the purpose of explanation, and does not limit the procedure of the present invention.

<Step S1>
First, in step S1, images and coordinates in the vicinity of the target position are collected using the data collection mobile object 2. The collected images are used to create approach movement learning data in the approach movement learning data creation unit 32a, and the data set of images and coordinates is used as registration learning data to be input to the registration learning preprocessing unit 32d.

Here, using Figures 5 and 6, we will explain how the data collection vehicle 2 collects each piece of data when the target position G is set in front of the door D.

FIG. 5 is a diagram illustrating a data collection range _A1 in which images necessary for creating learning data for approaching movement can be captured when a target position G is set in front of a door D. As is clear from this figure, when the data collection moving body 2 is within the sector-shaped (simplified to a triangular shape in FIG. 5) data collection range _A1 , it can capture an image of the door D, which serves as a landmark when approaching the target position G. Therefore, while moving the learning data collection moving body 2 in various ways within the data collection range _A1 , various images in which the door D (landmark) is captured, which are necessary for creating learning data for approaching movement, are collected. Note that, since a large number of similar images are not very useful as learning data, it is desirable to collect images evenly under various positions and conditions, such as the distance and angle from the door D or the brightness of the lighting. In addition, it is also possible to simplify the image collection work by taking a video when the data collection moving body 2 moves randomly within the data collection range _A1 , and later cutting out images from the video.

On the other hand, FIG. 6 is a diagram illustrating a data collection range _A2 for collecting a data set of images and coordinates, which is learning data for alignment, when a target position G is set in front of a door D. Here, the data collection mobile body 2 is moved around the target position G (for example, a square range of about 1 m surrounding the target position G) to collect a data set of images and coordinates at high density. For example, as shown in the balloon on the right side of the figure, the image acquisition unit 21 of the data collection mobile body 2 may be directed in various directions at each point of a high-density grid set around the target position G to collect a data set. In this case, the distance d of each grid and the step width Δθ of the data collection direction in each grid may be set in consideration of the target accuracy of the mobile body 1 using this embodiment. The learning data at the position between the grids may be interpolated by learning.

<Step S2>
In step S2, the approach movement command formula and the position alignment command formula are learned using the learning device 3. Details of this learning process will be described with reference to the flow chart shown in FIG.

First, in step S21, the approach movement learning data creation unit 32a acquires an image of a landmark (e.g., door D) specified via the object region input unit 31 from the image storage unit 33a, and creates approach movement learning data. Next, the approach movement learning pre-processing unit 32b reads the approach movement learning data from the approach movement learning data creation unit 32a, and the alignment learning pre-processing unit 32d reads the alignment learning data from the alignment learning data storage unit 33b.

Next, in step S22, the approach movement learning preprocessing unit 32b and the alignment learning preprocessing unit 32d perform preprocessing on each learning data to increase the variation of each learning data.

In step S23, the approach movement command formula learning unit 32c performs learning processing of the approach movement command formula, and the alignment command formula learning unit 32e performs learning processing of the alignment command formula.

Finally, in step S24, the learned approach movement command formula (e.g., formula 2, formula 3) is stored in the approach movement command formula storage unit 33c, and the alignment command formula (e.g., formula 1) is stored in the alignment command formula storage unit 33d.

<Step S3>
In step S3, a target position image including an object that can be a landmark is captured at a point that is to be the target position (final target position and waypoint) in the actual moving environment of the mobile body 1 using the image acquisition unit of the mobile body 1 or the data-collecting mobile body 2, or a digital camera carried by the system operator, and is registered in the target position image storage unit 14a of the mobile body 1. When the mobile body 1 moves autonomously, the target position image registered in the target position image storage unit 14a in this step becomes data indicating the target position.

Note that the imaging position in this step is different from the imaging position in step S1, but the surrounding environments of the imaging positions in step S1 and this step are similar to the extent that the command formulas learned in step S2 are useful. In other words, it is necessary to complete the learning process using the data collection vehicle 2 and the learning device 3 for an environment similar to the target position of the actual moving environment, and if the learning process for such an environment has not been completed, it is necessary to consider performing the processes of steps S1 and S2 for such an environment.

<Step S4>
In step S4, the moving body 1 is caused to autonomously move to the target position. If there is no change in the environment, the process of this step is repeatedly executed.

Figure 8 is a flowchart showing an example of this step in detail. Each process shown here may be executed in parallel, but for simplicity, the procedure is explained sequentially here.

Step S41: First, the movement switching unit 13a judges whether to switch the movement method of the moving body 1 to "alignment". For example, when the similarity between the current position image and the target position image reaches a certain value or more, when the landmark object reaches a threshold value or more in size, or when the relative position and orientation are continuously below the threshold value, it judges whether to switch to "alignment". If the result of the judgment is to switch to "alignment" (Yes), proceed to step S42. On the other hand, if "global movement" or "approaching movement" is to be continued (No), proceed to step S44.

Step S42: The moving body 1 moves toward the target position in response to a command from the position alignment command unit 11d. In this step, the moving body 1 moves to the target position based on the position alignment command formula. After moving for a predetermined time, the process proceeds to step S43.

Step S43: The movement switching unit 13a judges whether "alignment" has been completed, i.e., whether the moving body 1 has reached the target position. For example, if the relative position and orientation value to the target position calculated from the alignment command formula is equal to or less than the allowable value, it is judged that the target position has been reached and travel is terminated, and if not (No), the process proceeds to step S41 to continue "alignment" to the target position.

Step S44: The movement switching unit 13a judges whether to switch the movement method of the moving body 1 to "approach movement". For example, it judges whether an object (landmark) captured in a previously acquired target position image is captured in the current position image, and if a common object is captured (Yes), it switches to "approach movement", otherwise (No), it proceeds to step S46.

Step S45: The moving body 1 moves toward the target position by the approach movement command unit 13c. In this step, the moving body 1 moves toward the target position based on the approach movement command formula. After moving for a predetermined time, the process proceeds to step S41.

Step S46: The moving body is moved from the movement start position toward the target position by the global movement command unit 13b. After moving for a predetermined time, the process proceeds to step S41.

One feature of this embodiment is that the movement to the target position is divided into multiple steps: "global movement," "approach movement," and "alignment." In particular, what distinguishes this embodiment from conventional technology is the part in "approach movement" where an object in the target position image is used as a landmark in the current position image, focusing only on that part and estimating only the direction of movement, and the part in "alignment" where the entire current position image and target position image are used to estimate the relative position and orientation to the target position and move.

Due to this feature, when the mobile body 1 is far from the target position and there is little information about the target position in the current position image, it moves roughly focusing only on objects (landmarks) at the target position, and only when it approaches the target position and information about the target position increases in the current position image, does it accurately estimate the relative position and orientation to the target position using the entire current position image and the target position image. However, the range in which the position and orientation of the mobile body 1 can be accurately estimated is only a narrow range such as the example shown in Figure 6. This makes it possible to move more accurately to the target position and stop or pass through it even if there is a change in the driving environment that causes the number of objects other than landmarks to increase or decrease in the current position image (for example, an increase or decrease in the amount of traffic in a shopping mall depending on the day of the week).

The current position image used in step S42 is obtained by storing images from the past few shots and calculating the difference between them to select an image that does not include moving obstacles such as pedestrians, allowing for more accurate movement.

<Step S5>
Finally, in step S5, the operator judges whether re-learning in a new environment is necessary, i.e., whether it is necessary to actually drive in an environment dissimilar to the learned environment, and executes steps S1 and S2 again when it is judged necessary. On the other hand, if there is no such need, the process returns to step S3.

The following describes an example of the implementation of the navigation system 100 of this embodiment.

Figure 9 is a diagram showing an example of a navigation system 100 in which multiple moving bodies 1 exist. When the navigation system has multiple moving bodies 1A-1C as in this example, the data collection moving body 2 is equipped with a mechanism that can acquire a variety of data so that the image acquisition unit 21 can respond to the heights and mounting angles of the image acquisition units 11 of the moving bodies 1A-1C, and collects approach movement learning data and alignment learning data under installation conditions that match the various cameras of the moving bodies 1A-1C. Then, learning of the approach movement command formula and alignment command formula is performed based on the accumulated learning data.

The above-described configuration and procedures make it possible to provide a navigation device and navigation system for a moving object that is robust to changes in the driving environment and can move the moving object to a target position with high accuracy.

100 Navigation system 1, 1A to 1C Mobile body 11 Image acquisition section 12 Movement mechanism 13 Calculation unit 13a Movement switching section 13b Global movement command section 13c Approach movement command section 13d Alignment command section 13e Movement control section 14 Memory unit 14a Target position image memory section 14b Approach movement command formula memory section 14c Alignment command formula memory section 2 Data collection mobile body 21 Image acquisition section 22 Coordinate estimation sensor 23 Calculation unit 23a Alignment learning data creation section 23b Coordinate estimation section 24 Memory unit 24a Image memory section 24b Alignment learning data memory section 3 Learning device 31 Object area input section 32 Calculation unit 32a Approach movement learning data generation section 32b Approach movement learning pre-processing section 32c Approach movement command formula learning section 32d Alignment learning pre-processing unit 32e Alignment command formula learning unit 33 Storage unit 33a Image storage unit 33b Alignment learning data storage unit 33c Approach movement command formula storage unit 33d Alignment command formula storage unit

Claims (12)

A moving body that moves autonomously,
an image acquisition unit that captures a current position image;
a target position image storage unit that stores a target position image captured at the target position;
an approach movement command unit that generates a movement command for approaching and moving to the target position based on an approach movement command formula;
a position alignment command unit that generates a movement command to stop at the target position or a movement command to pass through the target position based on a position alignment command formula;
a movement switching unit that selects either the approach movement command unit or the alignment command unit according to an output result of the approach movement command formula or the alignment command formula;
A movement control unit that controls a movement mechanism based on the movement command;
It is equipped with
The approach movement command formula is composed of a formula that outputs an object position and an object type in an image when an image is input,
A moving body, wherein the position alignment command formula is composed of a formula that outputs a relative position and a relative orientation of the imaging positions when images captured at two positions are input. the target position image stored in the target position image storage unit is a target position image captured at a final target position and a target position image captured at a waypoint on the way to the final target position,
2. The moving body according to claim 1, wherein, during autonomous movement, the moving body autonomously moves to the final target position while switching the target position image used by the movement switching unit in the order of the waypoints. The moving body according to claim 1, characterized in that the positioning command unit calculates the relative position and relative orientation from the current position to the target position by comparing the current position image with the target position image, and outputs the movement command according to the calculation result. The moving body according to claim 1, characterized in that the approach movement command unit calculates the movement direction from the current position to the target position by comparing the current position image with the target position image, and outputs the movement command according to the calculation result. The movement switching unit is
When the similarity between the current position image and the target position image is equal to or greater than a predetermined threshold value,
When the same type of object is captured in the current position image and the target position image, and the size of the object in the current position image is equal to or larger than a predetermined threshold, or
When the relative position and orientation output by the position alignment command formula becomes equal to or smaller than a predetermined threshold value,
2. The moving body according to claim 1, wherein the position alignment command unit is selected in any one of the above cases. The movement switching unit is
The condition for selecting the alignment command unit is not satisfied,
6. The moving body according to claim 5, wherein the approach movement command unit is selected when the same type of object is captured in the target position image and the current position image. The moving body according to claim 6, characterized in that the movement switching unit selects the global movement command unit when the conditions for selecting the alignment command unit and the conditions for selecting the approach movement command unit are not met. a global movement command unit that generates a movement command for moving in the direction of a target position when performing global movement;
The moving body according to claim 1, characterized in that the movement switching unit selects either the global movement command unit, the approach movement command unit, or the alignment command unit, and switches from control by the global movement command unit to control by the approach movement command unit when a landmark object captured in the target position image is captured in the current position image. A navigation system for navigating a moving object according to any one of claims 1 to 8, comprising:
A navigation system according to claim 1, wherein the positioning command formula is obtained by subjecting learning data collected by a data-collecting mobile body to learning processing by a learning device. The navigation system of claim 9, characterized in that the learning data is a first image and a second image captured by the data-collecting mobile body, and the relative position and relative attitude from the capture position of the first image to the capture position of the second image. A navigation system for navigating a moving object according to any one of claims 1 to 8, comprising:
A navigation system according to claim 1, wherein the approach movement command formula is obtained by subjecting learning data collected by a data-collecting mobile body to learning processing by a learning device. The navigation system according to claim 11, characterized in that the learning data is a third image captured by the data-collecting mobile body and an area in which a specified object is present that is captured in the third image.

Images