JP2019197350A - Self-position estimation system, autonomous mobile system and self-position estimation method - Google Patents

Abstract

To allow self-position estimation robust against environmental variation.SOLUTION: A self-position estimation system includes: a camera 11 which captures an image of an environment surrounding a moving body; a state feature quantity acquisition unit 151 which acquires a state feature quantity during capturing the image; a learning device 2 which calculates an optimal value of a parameter for use in self-position estimation by learning processing on the basis of the state feature quantity and the image captured by the camera 11; and a self-position estimation unit 153 which uses the optimal value of the parameter to perform self-position estimation processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technology of a self-position estimation system, an autonomous movement system, and a self-position estimation method.

  In the self-position estimation method using a camera image, a self-position is estimated by comparing a reference image acquired in advance with an actual image acquired when the robot moves. One of the problems with such a technique is that the matching accuracy between the reference image and the actual image decreases when the surrounding environment such as brightness and weather changes. In addition, it is necessary to manually implement the response when the environmental conditions change, by making rules manually.

  As a countermeasure against such a problem, there is, for example, Patent Document 1. In Patent Document 1, “Robot 1 inputs image data obtained by imaging an object existing in the work environment by camera unit 101, and measures the distance from the object present in the work environment by laser range finder 107. In addition, i) the reprojection error calculation unit 103 detects the position of the landmark detected from the image data on the image and the work space included in the map information stored in the storage unit 106. A re-projection error between the position of the landmark and the re-projected position on the image data is calculated, and ii) the position error calculation unit calculates a registration error between the range data and the shape data included in the map information. Further, the optimization unit 109 determines an objective function including the reprojection error and the alignment error according to the surrounding environment of the robot 1, and determines the determined objective function as the maximum. It turned into a self-position estimation method of calculating to "mobile device and the mobile device its own position of the robot 1 is disclosed (see Abstract).

JP 2007-322138 A

  In the technique described in Patent Document 1, two self-position estimation methods must be combined. For this reason, the cost in terms of economy or time is high as compared with a case where the self-position estimation means is single, which is not preferable. In the technique described in Patent Document 1, a distance sensor must be used. However, since many distance sensors are more expensive than cameras, it is desirable to use only a camera.

  The present invention has been made in view of such a background, and an object of the present invention is to provide a self-position estimation system, an autonomous mobile system, and a self-position estimation method that are robust against environmental changes.

In order to solve the above-described problem, the present invention provides an imaging unit that captures an environment around a moving body, a state feature amount acquisition unit that acquires a state feature amount at the time of imaging by the imaging unit, and the state feature amount, Based on the image captured by the imaging unit, a learning processing unit for generating parameter information, which is information for deriving an optimum value of a parameter used for self-position estimation, by a learning process, and the imaging unit captured The parameter is calculated using an image of the current surrounding environment, the current state feature amount acquired by the state feature amount acquisition unit, and the parameter information, and self-position estimation processing is performed based on the calculated parameter. And a self-position estimation unit for performing.
Other solutions are described as appropriate in the embodiments.

  According to the present invention, it is possible to provide a self-position estimation system, an autonomous mobile system, and a self-position estimation method that are robust against environmental changes.

It is a figure which shows the structural example of the self-position estimation system which concerns on 1st Embodiment. It is a figure which shows the structural example of the learning apparatus in 1st Embodiment. It is a figure explaining the learning process of the parameter operation type used in 1st Embodiment. It is a figure which shows the procedure of the self-position estimation process performed in 1st Embodiment. It is a flowchart which shows the procedure of the whole process performed with the self-position estimation system of 1st Embodiment. It is a flowchart which shows the detailed procedure of this movement process performed in 1st Embodiment. It is a figure which shows the example of the self-position estimation system in which the several mobile body 1 exists. It is a figure which shows the structural example of the self-position estimation system which concerns on 2nd Embodiment. It is a figure which shows the structural example of the learning apparatus in 3rd Embodiment. It is FIG. (The 1) which shows the procedure which produces | generates the image in a rainy environment virtually. It is FIG. (2) which shows the procedure which produces | generates the image in a rainy environment virtually. FIG. 10 is a diagram (No. 1) illustrating a procedure for virtually generating an image captured by a camera with a narrow viewing angle. It is FIG. (2) which shows the procedure which produces | generates virtually the image imaged with the camera with a narrow viewing angle. It is a figure which shows the structural example of the self-position estimation system in 4th Embodiment.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described in detail with reference to the drawings as appropriate.

  When the surrounding environment such as brightness and weather changes, the self-position estimation system of this embodiment automatically changes the values of various parameters used for calculation of self-location to values suitable for the changed surrounding environment. adjust. By doing in this way, the self-position estimation system of this embodiment makes it possible to prevent deterioration of self-position estimation accuracy. In addition, the self-position estimation system according to the present embodiment uses a parameter operation formula for calculating a parameter corresponding to the environment in order to automatically calculate the parameter. Further, the parameter operation formula is generated by the learning device learning based on data collected in advance by a camera or the like.

[First Embodiment]
The first embodiment will be described with reference to FIGS.

(System configuration diagram and mobile 1)
FIG. 1 is a diagram illustrating a configuration example of a self-position estimation system Z according to the first embodiment.
The self-position estimation system (autonomous movement system) Z includes a moving body 1 and a learning device 2.
In addition, in FIG. 1, although the mobile body 1 is 1 unit | set, multiple units | sets may exist so that it may mention later.
The moving body 1 includes a camera (imaging unit) 11, a state information acquisition unit 12, a self-position estimation result display unit 13, an operation input unit 14, a calculation unit 15, and a storage unit 16.
The camera 11 acquires an image. The camera 11 is composed of one or more cameras. Each camera 11 includes an image sensor such as a CCD image sensor or a CMOS image sensor. Digital images obtained by imaging the outside world with these cameras 11 are output to the state feature quantity acquisition unit 151, the self-position estimation unit 153, and the learning data temporary storage unit 161.

In order to capture a wide range of driving environments, it is desirable to mount a plurality of cameras 11 or a camera 11 having a wide-angle lens or a super-wide-angle lens. Furthermore, when traveling in a dark place such as at night, it is desirable to install an infrared camera. In addition, when a plurality of cameras 11 are mounted, it is desirable to mount them at positions where the periphery of the moving body 1 can be imaged uniformly. Further, when the camera 11 having a wide-angle lens or a super-wide-angle lens is mounted, it is desirable that the camera 11 is mounted at the highest position of the moving body 1 and upward. By mounting the camera 11 in this manner, robustness can be improved.
Furthermore, it is desirable to fix the camera 11 so that it does not move during mounting. However, when a system for constantly grasping the posture of the camera 11 is installed, the posture of the camera 11 may not be fixed. Further, the image captured by the camera 11 may be a color image or a gray scale image.

The state information acquisition unit 12 acquires a sensing state (state information).
Specifically, the state information acquisition unit 12 acquires state information (hereinafter referred to as state information) from an image or the like acquired by the camera 11. The state is, for example, the brightness of the image. If the mobile unit 1 is connected to the Internet, the state information acquisition unit 12 acquires information about external elements such as weather as state information. The state information includes information related to internal elements of the moving body 1 such as the viewing angle of the camera 11. Information relating to the internal elements of the moving body 1 is obtained, for example, through input from a keyboard (that is, manual input by a user), description of a setting file, and the like.

  The self-position estimation result display unit 13 is a display or the like that displays the self-position estimation result. The self-position estimation result display unit 13 receives the current self-position information such as the position and posture of the moving body 1 from the self-position estimation unit 153, thereby displaying the position and posture of the moving body 1. As a display method, the position / orientation may be displayed with an arrow or a point on the surrounding map, or a numerical value calculated based on a specific origin may be displayed. In the example of FIG. 1, the self-position estimation result display unit 13 is mounted on the moving body 1, but may be mounted on a control PC or the like (not shown).

The motion input unit 14 is an interface for giving a movement command to the moving body 1 by a human, such as a handle, an accelerator pedal, or a joystick. In addition to these, the motion input unit 14 may be one in which a destination is designated on a map displayed on a tablet, or may be one in which coordinates of a destination are directly input by a keyboard.
The motion input unit 14 is used when manually operating the mobile body 1 to collect map data and learning data to be described later.
In addition, although the mobile body 1 performs autonomous movement, when collecting map data and learning data, a user performs operation using the operation input unit 14.

The calculation unit 15 includes a state feature amount acquisition unit 151, a parameter operation amount calculation unit 152, a self-position estimation unit 153, and an operation control unit 154.
A program is stored in the storage unit 16 of the mobile body 1. This program is loaded into a memory (not shown) and executed by a CPU (Central Processing Unit) (not shown). Thereby, the state feature amount acquisition unit 151, the parameter operation amount calculation unit 152, the self-position estimation unit 153, and the motion control unit 154 are realized.

The state feature amount acquisition unit 151 converts the state information obtained by the state information acquisition unit 12 into a state feature amount. The state feature amount is an index that quantitatively indicates the state by one or more numerical values. In learning in the learning device 2, a learning calculation is performed by using the state feature amount. For example, information expressed in a low dimension such as the height and viewing angle of the camera is used as a state feature amount as it is.
The state feature amount acquisition unit 151 also acquires a state feature amount from an image acquired from the camera 11. Note that information expressed in a high dimension such as an image is converted into information expressed in a low dimension such as an index of brightness (luminance) of the image and a ratio of obstacles in the screen. Incidentally, the brightness acquired by the state information acquisition unit 12 is the brightness of the surrounding environment.

The parameter operation amount calculation unit 152 calculates the parameter u based on the coefficient matrix W, the shift amount b, and the state feature amount x learned by the learning device 2. The processing performed by the parameter operation amount calculation unit 152 will be described later.
The self-position estimation unit 153 performs self-position estimation using the current image acquired by the camera 11 and the input parameter u.

  The motion control unit 154 controls the actuator of the mobile unit 1 according to the current self-position information such as the position and orientation of the mobile unit 1 input from the self-position estimation unit 153 and the movement command input from the operation input unit 141. .

The storage unit 16 includes a learning data temporary storage unit 161 and a parameter operation expression storage unit 162.
The learning data temporary storage unit 161 temporarily stores learning data including the image input from the camera 11 and the state feature amount acquired by the state feature amount acquisition unit 151. Note that the learning data is set data in which the time of the image and the state feature amount is synchronized.
The learning data temporarily stored in the learning data temporary storage unit 161 is copied to the learning data storage unit 221 of the learning device 2 periodically or when learning is performed. As a result, learning data is accumulated in the learning data accumulation unit 221 of the learning device 2. The learning device 2 performs learning using the learning data.

  The parameter operation expression storage unit 162 appropriately acquires the parameter operation expression learned by the learning process described later from the learning result storage unit 222 of the learning device 2. The parameter operation formula will be described later. As an example of this acquisition timing, before the mobile body 1 is operated, and the like.

The learning device 2 will be described later with reference to FIG.
In addition, communication between the mobile body 1 and the learning device 2 can be performed via a wired communication path such as an electric wire or an optical fiber, or wireless communication.

(Learning device 2)
FIG. 2 is a diagram illustrating a configuration example of the learning device 2 according to the first embodiment.
The learning device 2 includes a calculation unit 21 and a storage unit 22.
The calculation unit 21 includes a state feature amount input unit 211, an image data input unit 212, a reward calculation self-position estimation unit (learning processing unit) 213, a reward calculation unit (learning processing unit) 214, and parameter operation formula learning. Part (learning processing part) 215.
The storage unit 22 stores a program. When this program is loaded into a memory (not shown) and executed by a CPU (not shown), a state feature amount input unit 211, an image data input unit 212, a reward calculation self-position estimation unit 213, a reward calculation unit 214, The parameter operation formula learning unit 215 is embodied.

The storage unit 22 includes a learning data storage unit 221 and a learning result storage unit 222.
The learning data storage unit 221 stores the learning data stored in the learning data temporary storage unit 161 by the mobile unit 1 while the mobile unit 1 is moving. As described above, this accumulation is performed periodically, for example. In the learning process, image data of the learning data is input to the image data input unit 212. Of the learning data, the state feature amount is input to the state feature amount input unit 211.

  The state feature amount input unit 211 outputs the state feature amount input from the learning data storage unit 221 to the parameter operation formula learning unit 215. Similarly, the image data input unit 212 outputs the image data input from the learning data storage unit 221 to the reward calculation self-position estimation unit 213.

The reward calculation self-position estimation unit 213 estimates the self-position of the moving body 1 based on an image and a parameter acquired from the parameter operation formula learning unit 215 in order to calculate a reward used in a learning process described later. As a technique for self-position estimation, a feature point based technique or the like can be cited. As the feature point-based method, for example, the method described in Japanese Patent Application Laid-Open No. 2017-21427 is used.
Based on the processing result of the reward calculation self-position estimation unit 213, the reward calculation unit 214 calculates a reward that serves as a guide for minutely changing the weighting coefficient matrix W and the shift amount b.
The parameter operation formula learning unit 215 slightly changes the coefficient matrix W and the shift amount b based on the reward calculated by the reward calculation unit 214.

(Learning process)
Next, a learning process for learning the parameter operation formula will be described. Parameters are instructions that must be set manually in the self-position estimation. Parameters include, for example, the number of detected image feature points used when matching a map with an image, the range of use of an actual image in matching between image feature points constituting a map and image feature points in an actual image, etc. Is mentioned. By appropriately setting such parameters according to the state (brightness, etc.), it is possible to perform self-position estimation more accurately.
First, there is a formula (1) as an example of the parameter operation formula.

  u = Wx + b (1)

In Expression (1), x is a vector composed of state feature quantities. u is a vector composed of self-position estimation parameters. W is a weighting coefficient matrix of x. B is a coefficient vector representing the shift amount.
The learning process in the present embodiment is a process of adjusting the coefficient matrix W and the shift amount b so that the parameter u output when the state feature quantity x is input improves the self-position estimation accuracy. The mobile unit 1 can estimate an appropriate self-position estimation parameter for the state feature quantity x by the parameter operation formula using the learned coefficient matrix W and the shift amount b in the formula (1) at the time of self-position estimation. It becomes.

FIG. 3 is a diagram for explaining the learning process of the parameter operation formula used in the first embodiment.
First, the state feature quantity input unit 211 outputs the state feature quantity x to the parameter operation formula learning unit 215. The parameter operation equation learning unit 215 calculates the equation (1) using the initial value of the coefficient matrix W and the initial value of the shift amount b for the input state feature value x, and calculates the self-position estimation parameter u. . Then, the parameter operation formula learning unit 215 outputs the calculated parameter u to the reward calculation self-position estimation unit 213.

  The reward calculation self-position estimation unit 213 uses the parameter u and the image input from the image data input unit 212 to calculate an index (stability index) representing the stability of self-position estimation. Then, the reward calculation self-position estimation unit 213 outputs the calculated stability index to the reward calculation unit 214. The stability index will be described later.

  The reward calculation unit 214 calculates a reward for the parameter u based on the input stability index. Then, the reward calculation unit 214 outputs the calculated stability index to the parameter operation formula learning unit 215. Then, the parameter operation equation learning unit 215 changes the coefficient matrix W and the shift amount b by a minute amount based on the input reward. Then, the parameter operation equation learning unit 215 calculates the parameter u again using the parameter operation equation (equation (1)) using the coefficient matrix W and the shift amount b changed by a minute amount. Then, the parameter operation equation learning unit 215 outputs the calculated self-position estimation parameter u to the reward calculation self-position estimation unit 213.

Then, the reward is calculated in the same flow as the previous time.
The parameter operation equation learning unit 215 calculates a relationship between the coefficient matrix W, the change amount of the shift amount b, and the change amount of the reward based on the calculated current reward and the reward obtained in the previous calculation. Then, the parameter operation formula learning unit 215 slightly changes the coefficient matrix W and the shift amount b so that the reward becomes higher. Then, the same process is repeated to calculate a reward. The reward is set so as to increase as the parameter u approaches an appropriate value.

By repeating the above procedure, the parameter operation formula is continuously updated until the amount of change in the reward becomes a certain value or less. A series of flows shown in FIG. 3 is a learning process.
The learning process is performed for each set of learning data (that is, for each image).
The parameter u output by the parameter operation formula thus obtained is regarded as the optimum parameter for the state feature value x acquired in the environment. Then, a learning process is performed on all the learning data accumulated in various environments, thereby creating a parameter operation expression for estimating the optimum parameter u for various state feature quantities x. Then, the learned coefficient matrix W and the shift amount b created in this way are stored in the learning result storage unit 222.
The state feature amount x used for image processing, the learned coefficient matrix W, and the shift amount b are stored in the learning result storage unit 222 as a set of data.

The stability index includes, for example, the number of feature points constituting a map stored in advance and the number of feature points in the current image that are matched. It is determined that the greater the number of matched feature points, the higher the stability of self-position estimation.
In addition, as a reward, for example, if the index indicating the stability of self-position estimation is outside the target stability regulation range, a negative compensation is set, and if it is within the target stability regulation range, a positive compensation is given. Try to set. Such a prescribed range is set manually in advance.
When the parameter u becomes an appropriate value, the learning process is completed.

For example, when the surrounding environment (state feature amount x) is dark, the sensitivity (corresponding to the parameter u) for detecting a feature point is increased. In other words, in a bright environment, only the corners are used as feature amounts, but in a dark environment, there are cases where these feature amounts cannot acquire a sufficient amount for matching. In such a case, for example, the parameter u is adjusted such that a slightly rounded part is calculated as a feature amount.
By performing the learning process as shown in FIG. 3, efficient learning can be performed and a stable result can be obtained.

FIG. 4 is a diagram illustrating a procedure of self-position estimation processing performed in the first embodiment.
In FIG. 4, a self-position estimation process using the parameter operation formula (coefficient matrix W, shift amount b) calculated in the learning process shown in FIG. 3 will be described. .
The state feature amount acquisition unit 151 outputs the current state feature amount x to the parameter operation amount calculation unit 152.
The parameter operation amount calculation unit 152 acquires, from the parameter operation expression storage unit 162, the coefficient matrix W and the shift amount b corresponding to the acquired state feature amount x among the learned coefficient matrix W and the shift amount b shown in FIG. To do. The parameter operation amount calculation unit 152 acquires the state feature amount x from the state feature amount acquisition unit 151.
Next, the parameter operation amount calculation unit 152 substitutes the state feature amount x into the parameter operation equation obtained by applying the learned coefficient matrix W and the shift amount b to Equation (1). In this way, the parameter u is calculated. The parameter operation amount calculation unit 152 outputs the calculated parameter u to the self-position estimation unit 153. The self-position estimation unit 153 performs self-position estimation using the current image acquired by the camera 11 and the input parameter u.

(Overall processing)
FIG. 5 is a flowchart showing the procedure of the entire process performed in the self-position estimation system Z of the first embodiment. Reference is made to FIGS. 1 and 2 as appropriate.
When using the self-position estimation system Z of the present embodiment for the first time, it is necessary to collect learning data first. Therefore, learning data is first collected by manually operating the moving body 1 (S1), and the collected learning data is stored in the learning data temporary storage unit 161.
The learning data stored in the learning data temporary storage unit 161 is transmitted to the learning device 2 at a predetermined timing.
Then, the learning device 2 executes the learning process described above with reference to FIG. 3 using the transmitted learning data (S2). That is, the learning device 2 learns the parameter operation formula based on the learning data stored in the learning data storage unit 221. The processing in step S2 has been described with reference to FIG. The parameter operation expression learning unit 215 stores the parameter operation expression (coefficient matrix W and shift amount b) calculated as a result of the learning process in the learning result storage unit 222. The parameter operation formula stored in the learning result storage unit 222 is transmitted to the parameter operation formula storage unit 162 of the moving body 1 at a predetermined timing.

  Note that the fact that the learning process has ended may be displayed on a display unit (not shown) of the learning device 2, the self-position estimation result display unit 13 of the moving body 1, or the like. In this way, the user can confirm the end of learning.

  When learning is completed as described above, the moving body 1 can move while performing self-position estimation freely in the learned environment. Then, based on the stored learned parameter operation formula, the moving body 1 performs the movement process (S3). This movement process is that the moving body 1 moves while performing self-position estimation using the learned parameter operation formula. The process of step S3 will be described later with reference to FIG.

  When collecting the learning data in step S1 in FIG. 5, the user operates the operation input unit 14 such as a steering wheel, an accelerator, or a joystick to travel all within the assumed traveling environment. Then, learning data during traveling is collected. Then, the learning data is collected on the same travel route by changing the sensing state. For example, in a driving environment in which external light is inserted, it is desirable to collect learning data for morning, noon, and night. In addition, if it is outdoors, learning data in fine weather and rainy weather are collected as learning data in different sensing states. It is desirable that the state information acquisition unit 12 mounted on the moving body 1 collects learning data at various sensors that may be used and at various attachment positions.

(This move process)
FIG. 6 is a flowchart illustrating a detailed procedure of the main movement process (step S3 in FIG. 5) performed in the first embodiment. Reference is made to FIGS. 1 and 2 as appropriate.
First, the camera 11 captures an image by capturing the surrounding environment (S301).
Next, the state feature amount acquisition unit 151 acquires the state feature amount by the above-described method (S302).
Then, the parameter operation amount calculation unit 152 calculates the parameter u based on the parameter operation equation (coefficient matrix W and shift amount b) stored in the parameter operation equation storage unit 162 and the state feature amount (S303). . The processing in step S303 has been described with reference to FIG.

Subsequently, the self-position estimation unit 153 performs self-position estimation based on the calculated parameter u (S304). Since the process in step S304 is a known technique, its details are omitted.
Next, the self-position estimating unit 153 determines whether or not the self-position estimation is successful (S305). Although the self-position estimation success determination method includes the method described in Japanese Patent Application Laid-Open No. 2016-110576, other determination methods may be used.
If the result of step S305 is that the self-position estimation has failed (S305 → No), the image and state information at that time are stored in the learning data temporary storage unit 161 as new learning data (S311). And the calculation part 15 returns a process to step S301.

  As a result of step S305, when the self-position estimation is successful (S305 → Yes), the self-position estimation result display unit 13 displays the self-position estimation result (S321). In addition, after the process of step S305, “normal”, “unstable”, “calculation load”, and the like may be displayed on the self-position estimation result display unit 13 as a result of self-position estimation. By doing so, it becomes possible to cope with tracking lost.

Then, the operation control unit 154 determines whether or not the movement end condition is satisfied (S322). As the movement end condition, for example, the self-position estimation result of the moving body 1 is within a predetermined distance set in advance from the movement end target point. Alternatively, the movement end condition may be whether the operation control unit 154 has received a stop signal from the operation input unit 14. Alternatively, it may be possible that the self-position estimation has failed and it is necessary to collect learning data again.
As a result of step S322, when the operation end condition is satisfied (S322 → Yes), the calculation unit 15 ends the process.
As a result of step S322, when the operation end condition is not satisfied (S322 → No), for example, the operation control unit 154 is based on the destination specified on the map on the tablet or the like and the current self-position of the moving body 1. Moves the moving body 1 (S323). Specifically, the operation control unit 154 generates a route to the destination, and sends a signal that can follow the route to a moving mechanism (not shown) of the moving body 1.
And the calculation part 15 returns a process to step S301.

As shown in step S311, the learning data is stored in the learning data temporary storage unit 161 of the moving body 1 even when the moving process is being performed. Then, the accumulated learning data is transferred to the learning data storage unit 221 of the learning device 2 at a time when the moving process is not performed. Then, the learning device 2 performs the parameter operation equation learning step S201 each time, thereby updating the parameter operation equation. Specifically, when the mobile unit 1 is operated in a commercial facility, during the business hours of the commercial facility, the mobile unit 1 performs the main movement process and moves the data for which self-position estimation has failed as learning data. The data is stored in the learning data temporary storage unit 161 of the body 1. After the business hours, the learning data is transferred from the learning data temporary storage unit 161 of the mobile body 1 to the learning data storage unit 221 of the learning device 2. Then, outside of business hours, the learning device 2 performs a parameter operation expression learning process (S2), and newly sets a parameter operation expression (coefficient matrix W and shift amount b) corresponding to the state feature amount used in the learning process. To generate. Then, the parameter operation formula updated before the next business hours is transferred to the parameter operation formula storage unit 162 of the moving body 1. By doing in this way, the mobile body 1 can perform self-position estimation according to the latest sensing state.
If the acquired learning data is inappropriate for the learning process, the learning data is discarded.

(System example)
FIG. 7 is a diagram illustrating an example of a self-position estimation system Z in which a plurality of moving objects 1 exist.
When the self-position estimation system Z has a plurality of moving bodies 1A to 1C (1) as in the example of FIG. 7, learning data collected by the plurality of moving bodies 1A to 1C is stored in the common learning device 2. . Then, the learning device 2 performs learning based on the learning data.

  The self-position estimation system Z in the present embodiment learns the coefficient matrix W and the shift amount b so as to be appropriate parameters u using the state feature amount x. Then, the self-position estimation system Z calculates the parameter u using the learned coefficient matrix W and the shift amount b using the current state feature quantity x, and performs self-position estimation based on the calculated parameter u.

  With the above configuration, the mobile object 1 such as an automobile or a robot that performs self-position estimation can provide self-position estimation that is robust against environmental changes by a self-position estimation method using the camera 11.

  In addition, the self-position estimation system Z according to the present embodiment can perform efficient learning and movement of the moving body 1 by performing learning processing with a learning device different from the moving body 1.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG.
In the first embodiment, data collection and self-position estimation are performed using the same moving body 1. On the other hand, in the second embodiment, a case will be described in which learning data collection is manually performed using a moving body for data collection different from the moving body 1 used for self-position estimation.

(System Configuration)
FIG. 8 is a diagram illustrating a configuration example of the self-position estimation system Za according to the second embodiment.
As described above, the self-position estimation system Za differs greatly from the self-position estimation system Z shown in FIG. 1 in that a specialized data collection moving body 3 that collects learning data is provided.

The data collection moving body 3 includes a camera 31, a state information acquisition unit 32, an operation input unit 34, a calculation unit 35, and a storage unit 36.
The calculation unit 35 includes a state feature amount acquisition unit 351 and an operation control unit 354.
The storage unit 36 includes a learning data temporary storage unit 161.
The units 31, 32, 34, 351, 354 and the learning data temporary storage unit 161 in the data collection mobile unit 3 are the same as the units 11, 12, 14, 151, 154, 161 of the mobile unit 1 in FIG. Therefore, explanation here is omitted. However, the operation control unit 354 does not have a function of performing autonomous movement.

Since the learning device 2 has the same configuration as that shown in FIGS. 1 and 2, the description thereof is omitted here.
And the moving body 1a has the structure which abbreviate | omitted the learning data temporary storage part 161 from the moving body 1 shown in FIG. However, the learning data temporary storage unit 161 may be mounted on the moving body 1a.

(Operation)
The operation of the self-position estimation apparatus in the second embodiment is the same as that in the first embodiment, except that the learning data collection process (S1) in FIG.

It can be expected that the memory consumption of the mobile 1a can be reduced by performing the learning data collection process on the data collection mobile 3. As a result, the mobile body 1a can be designed and manufactured as a high quality product, while the data collection mobile body 3 can be designed with reduced design and design costs.
In addition, since the data collection mobile unit 3 specializes in data collection, it is possible to improve the operation efficiency of each of the data collection and the main movement process, and the failure of the mobile unit 1 (see FIG. 1) during data collection. Risk can also be reduced.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIGS. 9 to 11B.
In the third embodiment, a method for artificially generating data in various states based on collected data will be described. This makes it possible to acquire data that is actually difficult to collect.

Since the structure of the mobile body 1 is the same as that of the mobile body 1 shown in FIG. 1, illustration and description here are omitted.
(Learning device 2b)
FIG. 9 is a diagram illustrating a configuration example of the learning device 2b according to the third embodiment.
The learning device 2b shown in FIG. 9 is obtained by adding a learning data generation unit (image generation unit) 216 in the calculation unit 21b to the configuration of the learning device 2 shown in FIG. Other configurations have the same configuration as that of the learning device 2 shown in FIG.

  The learning data generation unit 216 in the learning device 2b first receives the learning data stored in the learning data storage unit 221. Then, learning data (image data and set data of state feature values) in various sensing states is generated virtually (artificially) for the received learning data. Further, the learning data generation unit 216 stores the learning data generated virtually (artificially) in the learning data storage unit 221. It is desirable that the learning data generation unit 216 automatically generate images in various sensing states when one image is input.

(Example of image virtually generated by learning data generation unit 216)
10A and 10B are diagrams illustrating a procedure for virtually generating an image 402 in a rainy environment.
FIG. 10A shows an image (original image 401) captured in fine weather.
10B is generated virtually (artificially) by the learning data generation unit 216 from the original image 401 in FIG. 10A. Here, the learning data generation unit 216 generates an image 402 whose field of view is blocked by adding raindrops or the like after reducing the contrast with respect to the original image 401 of FIG. 10A.

FIGS. 11A and 11B are diagrams illustrating a procedure for virtually generating an image 403 captured by the camera 11 having a narrow viewing angle.
An original image 401 shown in FIG. 11A is the same image as the original image 401 shown in FIG. 10A.
In other words, an image 403 shown in FIG. 11B is generated virtually (artificially) by the learning data generation unit 216 from the original image 401 in FIG. 11A.
The learning data generation unit 216 generates an image 403 obtained by performing trimming for narrowing the image area on the original image 401 illustrated in FIG. 11A.

  10B and 11B show an image 402 in a rainy environment and an image 403 captured by a camera with a narrow viewing angle. In addition to this, image generation when the sensing state is dark / bright or the entire image For example, adjusting the contrast. The example given here is an example, and an image under another state may be virtually generated.

(Overall processing)
Next, the operation of the self-position estimation system Z performed in the third embodiment will be described.
In step S <b> 1 of FIG. 5, after the mobile unit 1 has collected learning data in the assumed driving environment, the learning data generation unit 216 virtually generates learning data. After that, the parameter operation expression learning process (S2) and the main movement process (S3) are performed.

  According to the third embodiment, learning data is virtually generated, so that the labor for collecting learning data can be reduced. In addition, learning data in a sensing state that is actually difficult to collect can be used.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described with reference to FIG.
In the fourth embodiment, in addition to the configuration of the second embodiment, the environmental feature amount is visualized by a learning data collection process (S1 in FIG. 3). An embodiment that realizes efficient collection of learning data by preventing duplicate collection of useless data in this way will be described.

FIG. 12 is a diagram illustrating a configuration example of the self-position estimation system Zc in the fourth embodiment.
The self-position estimation system Zc shown in FIG. 12 is an extension of the configuration of the self-position estimation system Za shown in FIG.
Here, since the mobile body 1a and the learning device 2 have the same configuration as that shown in FIG. 8, the same reference numerals are given and description thereof is omitted.
The data collection mobile unit 3c has a configuration in which a state feature amount display unit (display unit) 37 and a state feature amount storage unit 361 are added to the data collection mobile unit 3 of FIG. Other configurations are the same as those of the data collection moving body 3 of FIG.
The state feature amount storage unit 361 stores the state feature amount of the learning data from the learning data temporary storage unit 161. The state feature amount storage unit 361 stores state feature amounts of learning data collected by the data collection mobile unit 3c in the past. Then, the state feature amount accumulated in the past is output to the state feature amount display unit 37. In order to reduce the memory of the storage unit 36c, the state feature amount of the collected learning data is not directly stored, but may be stored in the form of a statistic such as an average or variance, or a distribution.

  The state feature amount display unit 37 receives information feature amounts of learning data collected in the past from the state feature amount storage unit 361. In addition, the state feature quantity currently being sensed by the camera 11 is received from the state feature quantity acquisition unit 351. Then, the state feature amount display unit 37 compares and visualizes the environmental feature amount at the time of past data collection and the environmental feature amount currently being sensed. As a visualization method, for example, a method of superimposing a current value on data collected in the past and distributing the data may be used, but another method may be used.

According to the fourth embodiment, it is possible to collect data while comparing the data collected in the past with the current value, and therefore it is possible to prevent redundant collection of unnecessary data. As a result, efficient collection of learning data can be realized.
Note that the state feature value display unit 37 may display the learning data of the overlapping sensing state, and the user may discard unnecessary learning data while viewing the display.

[Fifth Embodiment]
The state information acquisition unit 12 may use an external sensor such as an illuminometer, or may be manually input from the calculation unit 15 such as the height (attachment position) of the camera 11. By doing in this way, a state feature-value can be increased and the precision of self-position estimation can be improved.

  The self-position estimation system Z of the present embodiment can be applied to an automatic driving system, a mobile phone position estimation service, a virtual reality device using self-position estimation, shoes for calculating a moving distance based on the self-position, and a mobile device. It is.

  The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Further, each of the above-described configurations, functions, units 151 to 154, 211 to 216, 351, 354, storage units 16, 22, 36, etc. can be realized by designing a part or all of them by, for example, an integrated circuit. It may be realized with. Further, each configuration, function, and the like described above may be realized by software by a processor such as a CPU interpreting and executing a program that realizes each function. In addition to storing information such as programs, tables, and files for realizing each function in an HD (Hard Disk), a memory, a recording device such as an SSD (Solid State Drive), an IC (Integrated Circuit) card, It can be stored in a recording medium such as an SD (Secure Digital) card or a DVD (Digital Versatile Disc).
In each embodiment, control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it can be considered that almost all configurations are connected to each other.

1, 1a, 1A to 1C Moving object 2, 2b Learning device 3, 3c Moving object for data collection 11, 31 Camera (imaging unit)
37 State feature amount display part (display part)
151, 351 State feature value acquisition unit 153 Self-position estimation unit 213 Reward calculation self-position estimation unit (learning processing unit)
214 Reward calculator (learning processor)
215 Parameter operation expression learning unit (learning processing unit)
216 Learning data generation unit (image generation unit)
Z, Za, Zc Self-position estimation system (autonomous mobile system)

Claims (9)

An imaging unit that images the surrounding environment of the moving body;
A state feature amount acquisition unit for acquiring a state feature amount at the time of imaging by the imaging unit;
A learning processing unit that generates, by learning processing, parameter information that is information for deriving an optimum value of a parameter used for self-position estimation based on the state feature amount and an image captured by the imaging unit;
The parameter is calculated using an image of the current surrounding environment captured by the imaging unit, the current state feature amount acquired by the state feature amount acquisition unit, and the parameter information, and the calculated parameter is used as a basis. A self-position estimation unit that performs self-position estimation processing,
A self-position estimation system comprising: An imaging unit that images the surrounding environment of the moving body;
A state feature amount acquisition unit for acquiring a state feature amount at the time of imaging by the imaging unit;
A learning processing unit that generates, by learning processing, parameter information that is information for deriving an optimum value of a parameter used for self-position estimation based on the state feature amount and an image captured by the imaging unit;
The parameter is calculated using an image of the current surrounding environment captured by the imaging unit, the current state feature amount acquired by the state feature amount acquisition unit, and the parameter information, and the calculated parameter is used as a basis. A self-position estimation unit that performs self-position estimation processing,
An operation control unit that moves based on a result of the self-position estimation process;
An autonomous mobile system characterized by comprising: The imaging unit, the state feature quantity acquisition unit, the self-position estimation unit, and the motion control unit are provided in a moving body,
The autonomous mobile system according to claim 2, wherein the learning processing unit is provided in a learning device. The imaging unit and the state feature quantity acquisition unit are provided in a data collection moving body,
The self-position estimating unit and the motion control unit are provided in a moving body,
The autonomous mobile system according to claim 2, wherein the learning processing unit is provided in a learning device. The mobile body for data collection includes
The autonomous mobile system according to claim 4, further comprising a display unit that displays information on the collected state feature amount. An image generation unit that generates images with different conditions from a predetermined image,
The state feature acquisition unit
The autonomous mobile system according to claim 2, wherein the state feature amount is acquired from the image generated by the image generation unit. The autonomous mobile system according to claim 2, wherein the state feature quantity acquisition unit includes an illuminometer. The self-position estimating unit
The autonomous mobile system according to claim 2, wherein the image and the state feature amount that have failed in the self-position estimation are sent to the learning processing unit. The imaging unit performs a first imaging step of imaging the surrounding environment of the moving body,
The state feature amount acquisition unit performs a first state feature amount acquisition step of acquiring a state feature amount at the time of imaging by the first imaging step,
A learning processing unit generates parameter information, which is information for deriving an optimum value of a parameter used for self-position estimation, based on the state feature amount and the image captured by the image capturing unit by a learning process. Perform learning process steps,
The imaging unit performs a second imaging step of imaging the surrounding environment of the current moving body,
The state feature amount acquisition unit performs a second state feature amount acquisition step of acquiring a state feature amount at the time of imaging by the second imaging step,
The self-position estimation unit uses the current surrounding environment image captured in the second imaging step, the current state feature amount acquired in the second state feature amount acquisition step, and the parameter information. A self-position estimation step of calculating the parameter and performing a self-position estimation process based on the calculated parameter.

Images