JP7153505B2 - moving body - Google Patents

Description

TECHNICAL FIELD The present invention relates to a moving object that acquires its current position using a captured image captured by a capturing unit attached to a manipulator.

2. Description of the Related Art Conventionally, a mobile robot having a manipulator provided with an imaging device is known (see Patent Document 1). The mobile robot can run by itself using the travel information marks captured by the imaging device of the manipulator.

Japanese Utility Model Laid-Open No. 60-066486

However, in a moving object that moves autonomously using captured images, there is a problem that highly accurate movement cannot be realized if information used for movement control is not appropriately captured. For example, there is a problem that it may not be possible to perform appropriate movement when photographing is performed against the light.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and is to improve the accuracy of the current position of a moving body that obtains the current position using a photographed image obtained by a photographing unit attached to a manipulator. With the goal.

In order to achieve the above object, a mobile body according to the present invention is a mobile body that moves autonomously, comprising: a manipulator; a position estimating unit for estimating the position of the imaging unit using a map in the moving area of the moving object; and a current position acquisition for obtaining the current position of the moving object using the position of the imaging unit estimated by the position estimating unit. a movement mechanism for moving a moving object; a movement control unit for controlling the movement mechanism using the current position; a manipulator control unit for controlling a manipulator; and a photographing direction control unit for changing the photographing direction of the photographing unit by the manipulator control unit so that a photographed image suitable for position estimation is photographed using matching information. It is a thing.
With such a configuration, by controlling the shooting direction using matching information, it becomes possible to acquire a shot image that is more suitable for position estimation, and as a result, the accuracy of the current position can be improved. will be able to

In addition, in the moving object according to the present invention, the matching information includes the representative value of the luminance of the captured image, and the photographing direction control unit determines that the representative value is The shooting direction may be changed so that the image becomes smaller.
With such a configuration, for example, when the captured image is overexposed and the feature points for position estimation cannot be acquired properly, the shooting direction is changed so as to obtain a more appropriate exposure. By doing so, it becomes possible to perform position estimation with higher accuracy.

Further, in the moving body according to the present invention, the conformity information includes turning information indicating whether or not the immediately following movement by the moving mechanism is turning. is a turn, the photographing direction may be changed so that the optical axis of the photographing unit approaches the turning axis direction.
With such a configuration, for example, the photographing unit can photograph the upper side and the lower side when the moving body is turning. In position estimation using captured images, tracking may not be possible during turning. tracking can be done properly.

In addition, in the moving body according to the present invention, the matching information includes ratio information indicating the ratio of matching between the feature points extracted from the captured image and the feature points included in the map. If the ratio indicated by the information is smaller than a predetermined threshold, the shooting direction may be changed so that the ratio becomes larger.
With such a configuration, for example, when an obstacle or the like that temporarily exists around the moving object makes it impossible to perform appropriate position estimation, such an obstacle or the like is avoided for shooting. In addition, the shooting direction can be changed, and position estimation with higher accuracy can be realized.

Further, in the moving object according to the present invention, the shooting direction control unit may change the shooting direction so that the position of the shooting unit is higher when the ratio indicated by the ratio information is smaller than a predetermined threshold.
With such a configuration, by changing the shooting direction so as to shoot from a higher place, the possibility of realizing shooting while avoiding obstacles or the like increases.

According to the mobile object of the present invention, it becomes possible to acquire a photographed image more suitable for position estimation, thereby improving the accuracy of the current position.

Schematic diagram showing the appearance of a moving body according to an embodiment of the present invention FIG. 2 is a block diagram showing the configuration of a moving object according to the same embodiment; 4 is a flow chart showing the operation of a moving object according to the same embodiment; FIG. 4 is a diagram for explaining turning of a moving body according to the same embodiment; FIG. 4 is a diagram for explaining matching between feature points of a map and feature points of a photographed image in the same embodiment; FIG. 4 is a diagram for explaining a specific example of the operation of a moving object according to the same embodiment;

Hereinafter, a moving body according to the present invention will be described using embodiments. In the following embodiments, constituent elements and steps with the same reference numerals are the same or correspond to each other, and repetitive description may be omitted. The moving body according to the present embodiment performs position estimation using a photographed image acquired by an imaging unit attached to a manipulator, and uses matching information indicating whether the photographed image is suitable for position estimation. By doing so, the photographing direction is controlled.

FIG. 1 is a schematic diagram showing the appearance of a mobile body 1 according to this embodiment, and FIG. 2 is a block diagram showing the functional configuration of the mobile body 1. As shown in FIG. The moving object 1 according to the present embodiment moves autonomously, and includes a manipulator 11, a manipulator control unit 12, an imaging unit 13, a position estimation unit 14, a current position acquisition unit 15, and a moving mechanism. 16 , a movement control unit 17 , an acquisition unit 18 , and a shooting direction control unit 19 . It should be noted that the autonomous movement of the mobile body 1 may mean that the mobile body 1 moves to the destination based on its own judgment, instead of moving according to an operation instruction received from a user or the like. The destination may, for example, be manually determined or automatically determined. Also, the movement to the destination may or may not, for example, be along a movement route. Further, moving to the destination by one's own judgment may be, for example, moving to the destination by the moving body 1 judging the traveling direction, movement, stop, or the like. Alternatively, for example, the moving body 1 may move so as not to collide with an obstacle. The mobile body 1 may be, for example, a trolley or a moving robot. The robot may be, for example, an entertainment robot, a surveillance robot, a transport robot, a cleaning robot, or a robot that shoots moving images or still images. , a welding robot, a painting robot, an agricultural chemical spraying robot, an assembly robot, or any other robot.

The manipulator 11 may have a plurality of arms connected by joints driven by motors. The tip of the manipulator 11 may be provided with an end effector corresponding to the work performed by the manipulator 11, such as welding, gripping an object to be conveyed, painting, and assembly. The manipulator 11 may be, for example, a manipulator of a vertical articulated robot or a manipulator of a horizontal articulated robot. Moreover, the number of axes of the manipulator 11 is not limited. As shown in FIG. 1 , the base end side of the manipulator 11 may be fixed to the vehicle body 1 a of the mobile body 1 . Also, the manipulator 11 may have a shaft on the base end side that can rotate the entire manipulator 11 .

The manipulator controller 12 controls the manipulator 11 . The manipulator control unit 12 may, for example, control the position and orientation of the hand of the manipulator 11, or may control the end effector of the manipulator 11. FIG. The manipulator control unit 12 may control the manipulator 11 via, for example, a servo controller that controls each motor of the manipulator 11 .

The photographing unit 13 acquires a photographed image of the surroundings of the moving body 1 . The imaging unit 13 is attached to the manipulator 11 . Although the position of the manipulator 11 to which the photographing unit 13 is attached does not matter, the photographing unit 13 may be attached to the tip of the manipulator 11, for example. It is assumed that the position at which the photographing unit 13 is mounted on the manipulator 11 is already known. Further, the photographing unit 13 may be attached to the manipulator 11 so as to photograph the front side of the end effector, for example. By doing so, it becomes possible to use the photographed image acquired by the photographing unit 13 even when working with the end effector. Since the imaging unit 13 is attached to the manipulator 11 in this manner, the imaging direction is changed according to the change in the position and posture of the manipulator 11 . The photographing unit 13 may acquire a photographed image using, for example, an image sensor such as a CCD image sensor or a CMOS image sensor. Also, the imaging unit 13 may include an optical system such as a lens for forming an image on the image sensor. Further, the imaging unit 13 may be monocular or binocular (stereo camera). In this embodiment, a case where the imaging unit 13 is monocular will be mainly described. The imaging unit 13 normally captures moving images, that is, acquires continuous image frames. Since a photographed image is acquired in order to acquire the current position of the mobile body 1 that is moving, it is preferable that the frame rate is sufficiently high relative to the moving speed. For example, the frame rate may be approximately 30 fps, or the like. In addition, except when the shooting direction control unit 19 controls the shooting direction according to compatibility information, which will be described later, the shooting unit 13 may, for example, acquire a shot image in the traveling direction. Alternatively, a photographed image may be acquired in a horizontal direction with respect to the traveling direction.

The position estimating unit 14 estimates the position of the photographing unit 13 using the photographed image acquired by the photographing unit 13 and the map in the movement area of the moving body 1 . With this position estimation, the position of the imaging unit 13 in the global coordinate system is estimated. This position may include orientation. The position estimation unit 14 may extract feature points from the captured image and perform position estimation by matching the extracted feature points with feature points included in the map. The map is also called an environment map, and may be stored in a recording medium (not shown), for example. The feature points may be, for example, FAST keypoints, SIFT keypoints, SURF keypoints, or other feature points used in position estimation using captured images. The matching may be performed by calculating the degree of similarity between feature descriptors of feature points identified in the captured image and feature descriptors of feature points included in the map. Since position estimation using such captured images is already known as visual-SLAM (Simultaneous Localization and Mapping), detailed description thereof will be omitted. Note that the position estimation unit 14 may or may not update the map using the feature points specified in the captured image. In this embodiment, a case where the position estimating unit 14 performs position estimation by visual-SLAM, that is, a case where the map is also updated will be mainly described. Also, when the visual-SLAM is ORB-SLAM, the feature points included in the map may be, for example, map points or feature points included in keyframes. For FAST keypoints, SIFT keypoints, SURF keypoints, feature points in ORB-SLAM, position estimation (tracking) in ORB-SLAM, and map updating (local mapping), see the following documents 1 to 5, respectively. want to be

Reference 1: Edward Rosten, Tom Drummond, "Machine learning for high-speed corner detection", in 9th European Conference on Computer Vision, vol. 1, 2006, pp. 430-443.
Reference 2: D. Lowe, "Distinctive image features from scale-invariant keypoints", Int. Journal of Computer Vision (2004), 60(2), pp. 91-110.
Reference 3: H. Bay, A. Ess, T. Tuytelaars, L. Van Gool, "SURF: Speeded Up Robust Features", Computer Vision and Image Understanding (CVIU) (2008), 110(3), pp. 346- 359.
Reference 4: Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski, "ORB: An efficient alternative to SIFT or SURF", ICCV 2011, pp. 2564-2571.
Reference 5: R. Mur-Artal, JM Montiel, JD Tardos, "ORB-SLAM: A Versatile and Accurate Monocular SLAM System", IEEE Transactions on Robotics 31 (2015), pp. 1147-1163.

The current position acquisition unit 15 acquires the current position of the moving body 1 using the position of the imaging unit 13 estimated by the position estimation unit 14 . The angle of each joint of the manipulator 11 is managed by the manipulator control unit 12, and the mounting position of the imaging unit 13 on the manipulator 11 is known. The position of the part 13 can be known. The position of the imaging unit 13 may be, for example, the position in the local coordinate system of the moving body 1 . Therefore, using the position of the imaging unit 13 in the global coordinate system acquired by the position estimating unit 14 and the position of the imaging unit 13 in the mobile object 1, the current position acquisition unit 15 obtains the position of the mobile object 1 in the global coordinate system. You can get the current position. The current position may be, for example, a position in the global coordinate system of a predetermined location of the moving body 1 (for example, a position of the center of gravity, a position predetermined by the designer, etc.). This current position may also include orientation.

The moving mechanism 16 moves the moving body 1 . The moving mechanism 16 may, for example, move the moving body 1 in all directions, or may not. Being able to move in all directions means being able to move in any direction. The moving mechanism 16 may have, for example, a traveling portion (eg, wheels) and a driving means (eg, a motor, an engine, etc.) that drives the traveling portion. Further, the moving mechanism 16 may have a mechanism capable of acquiring the speed of wheels, for example, an encoder. In addition, when the moving mechanism 16 can move the moving body 1 in all directions, the traveling part may be an omnidirectional moving wheel (for example, an omni wheel, a mecanum wheel, etc.). Since a known mechanism can be used as the moving mechanism 16, detailed description thereof will be omitted.

The movement control unit 17 controls movement of the moving body 1 by controlling the movement mechanism 16 using the current position acquired by the current position acquisition unit 15 . The control of the movement may be control of the direction of movement of the mobile body 1, start/stop of movement, and the like. For example, when a movement route is set, the movement control unit 17 may control the movement mechanism 16 so that the moving body 1 moves along the movement route. More specifically, the movement control section 17 may control the movement mechanism 16 so that the current position acquired by the current position acquisition section 15 is along the movement route. Further, the movement control unit 17 may control movement using a map. Since control of the movement mechanism 16 by the movement control unit 17 is well known, detailed description thereof will be omitted.

The acquisition unit 18 acquires matching information indicating whether the captured image is suitable for position estimation. Whether or not a captured image is suitable for position estimation may be considered to mean whether or not the captured image can be used to estimate a position with higher accuracy. For example, as will be described later, the matching information may be acquired according to the photographed image, the content of the movement control immediately after the moving body 1, the matching situation between the feature points of the photographed image and the feature points of the map, and the like. .

For example, if the captured image is overexposed, that is, if the captured image has blown out highlights, it becomes difficult to identify the feature points in the captured image. is not suitable for Therefore, the matching information may include the representative value of the brightness of the captured image. The representative value may be, for example, an average value, a median value, or a minimum value. For example, the obtaining unit 18 may obtain the luminance value of each pixel included in the captured image and obtain a representative value of the plurality of obtained luminance values. If the representative value of the luminance values of the captured image is greater than a predetermined threshold value, the captured image may be determined not to be suitable for position estimation. This is because the captured image is considered to be overexposed.

Further, for example, when the moving body 1 turns, if the image is taken in the horizontal direction, it may not be possible to perform appropriate tracking in estimating the position using the imaged image. Therefore, the horizontal image captured when the moving body 1 turns is not suitable for position estimation. Accordingly, the fit information may include turning information indicating whether the subsequent movement by the moving mechanism 16 is turning. The turning information may be flag-like information indicating whether or not the immediately following movement is turning. It may be information indicating the speed of change (angular velocity) or the degree of change in the traveling direction (angle, etc.).

Specifically, in the future movement of the moving body 1, the acquisition unit 18 determines that the traveling direction is at a predetermined threshold angle (e.g., , 80 degrees, 90 degrees, 120 degrees, 150 degrees, etc.), it may be determined that the subsequent movement is a turn, and turn information indicating that the immediately following movement will be a turn may be acquired. .

FIGS. 4A and 4B are diagrams showing changes in the traveling direction of the moving body 1 when viewed from above. For example, when the future traveling direction of the moving body 1 changes in order as indicated by arrows A1 to A5 shown in FIG. You may acquire turning information which shows. It should be noted that if such a turn is made within a predetermined time interval, it may be determined that the future movement is a turn. Also, FIG. 4(a) shows a turn with a turning radius of 0, but as shown in FIG. 4(b), the above conditions are satisfied even if the turning radius is not 0. If so, it may be determined that the upcoming movement is a turn. This is because, if the traveling direction changes by a predetermined angle or more within a predetermined time interval, tracking in position estimation using horizontal images becomes difficult regardless of the turning radius. Note that FIG. 4B shows that the moving body 1 moves along the route R1 and the direction of travel changes in order as indicated by arrows A6 to A8. Further, the acquisition unit 18 may acquire, for example, the traveling direction of future movement from the movement control unit 17, or may calculate the movement route, the current position, or the like. In addition, since the movement control unit 17 performs movement control according to the current position, it may have only information on movement during the most recent predetermined period. In such a case, the acquisition unit 18 determines that the future movement is a turn when the change in the angle of the direction of travel in the future is equal to or greater than the threshold angle in the information held by the movement control unit 17. You may Further, if the turn information indicates that the vehicle will turn immediately, it may be determined that the captured image is not suitable for position estimation. This is because a horizontal captured image captured during turning is not suitable for tracking.

Further, for example, when an obstacle such as a human exists between an object corresponding to a feature point on the map used for position estimation and the photographing unit 13, the feature point extracted from the photographed image and the map , the degree of matching with the feature points included in is degraded, and as a result, the accuracy of position estimation is degraded. Therefore, the match information may include ratio information indicating the ratio of matching between the feature points extracted from the captured image and the feature points included in the map. FIG. 5 is a diagram showing a matching relationship between feature points extracted from a captured image and feature points included in a map.

For example, as shown in FIG. 5A, it is assumed that the map includes feature points A to D and the feature points a to d are specified in the captured image. Further, by matching using feature amounts (feature descriptors) between the feature points specified in the captured image and the feature points included in the map, the feature points a to d of the captured image are matched to the feature points A to d of the map, respectively. Suppose we are to deal with D. The matching may be performed by the position estimator 14 for position estimation by visual-SLAM, for example. In the case of FIG. 5(a), four feature points are extracted from the captured image, and all of the feature points are matched with feature points included in the map, so the ratio information is 1 or 100%. .

On the other hand, as shown in FIG. 5B, assume that the map includes feature points A to D, and feature points a and d are specified in the captured image. Further, the feature points a and d of the photographed image correspond to the feature points A and D of the map, respectively, by matching using the feature amounts of the feature points specified in the photographed image and the feature points included in the map. Suppose it became In addition, in FIG. 5B, the map also includes feature points B and C. As shown in FIG. Therefore, originally, the feature points b and c corresponding to the feature points B and C of the map should be extracted from the photographed image, but the extraction is not performed. Then, in the photographed image, the feature points b and c are the feature points of missing extraction. Note that, for example, when a feature point having a feature amount similar to that of the feature point B is not extracted in the vicinity of the position where the feature point B included in the map is projected onto the captured image, the feature point B is extracted. It may be determined to be a leak. In this case, the ratio information may be calculated by the following equation. In addition, in the case of percentage, the value of the following formula may be multiplied by 100. In the case of FIG. 5B, two feature points that match the feature points of the map are extracted from the photographed image, and there are two feature points that are not extracted, so the ratio information is 0.5 or 50. %.
Ratio information = N1/(N1+N2)
However, N1 is the number of feature points that match feature points included in the map among the feature points extracted from the captured image, and N2 is the number of missing feature points for the captured image.

Also, as shown in FIG. 5C, when there is a feature point that has not been extracted, other feature points e and f may be extracted from the captured image. For example, in ORB-SLAM, the number of feature points (FAST keypoints) to be extracted from a captured image is predetermined, so feature points are extracted until the predetermined number is reached. In such a case, for example, if the feature points e and f are acquired from an obstacle such as a person existing between the object corresponding to the feature points B and C on the map and the photographing unit 13, the photographing There is no feature point on the map corresponding to the feature points e and f of the image. In such a case, feature points a and d match feature points A and D on the map, but feature points e and f do not match feature points on the map. In this case, the ratio information may be calculated by the following equation. In addition, in the case of percentage, the value of the following formula may be multiplied by 100. In the case of FIG. 5C, four feature points are extracted from the captured image, and two feature points are matched with feature points on the map, so the ratio information is 0.5 or 50%.
Ratio information = N1/N3
However, N1 is as described above, and N3 is the number of feature points extracted from the captured image.

In this way, the acquisition unit 18 obtains the number (N3) of feature points extracted from the captured image, and the number (N1 ), the number (N2) of feature points that are not extracted regarding the captured image, or the like may be used to obtain the ratio information. The ratio information may be calculated by either of the above two formulas, or may be calculated by another formula. Also, if the ratio indicated by the ratio information is smaller than a predetermined threshold, it may be determined that the captured image is not suitable for position estimation. This is because there is a possibility that an obstacle that is not included in the map exists in front of the photographing unit 13 in the photographing direction.

Here, the description has been given of the case where the ratio information is lowered due to the existence of a movable obstacle such as a human being in the movement area of the moving body 1. However, the ratio information is lowered due to other factors, such as It can also occur due to shooting with backlight. Therefore, by controlling the photographing direction using the ratio information, it is possible to reduce the influence of backlight and the like.

The photographing direction control unit 19 uses the matching information acquired by the acquiring unit 18 to change the photographing direction of the photographing unit 13 so that a photographed image suitable for position estimation is photographed. The photographing direction is changed by the photographing direction control section 19 controlling the manipulator control section 12 . When performing control to change the imaging direction, for example, the imaging direction control unit 19 determines the imaging direction of the imaging unit 13 after the change, and the position of the manipulator 11 to which the imaging unit 13 is attached changes the imaging direction. The manipulator control unit 12 may be instructed to set the direction. The manipulator control unit 12 calculates, by inverse kinematics, the angle of each joint so that the part of the manipulator 11 to which the imaging unit 13 is attached is in a predetermined imaging direction. You may control the manipulator 11 so that it may become an angle. The change in the imaging direction may be, for example, a change in the direction of the optical axis of the imaging unit 13, or a change in the position of the imaging unit 13, such as when the optical axis of the imaging unit 13 is translated. good too.

When the matching information includes the representative value of the luminance of the captured image, the photographing direction control unit 19 controls the photographing so that the representative value of the luminance of the captured image becomes smaller when the representative value of the luminance of the captured image is larger than a predetermined threshold. You can change the direction. Note that when the representative value of the brightness of the captured image is smaller than the predetermined threshold value, the shooting direction control unit 19 does not have to control the shooting direction according to the representative value of the brightness of the captured image. Further, when the representative value of the brightness of the captured image is the same as the predetermined threshold value, the shooting direction control section 19 may or may not control the shooting direction. The change in the imaging direction may be, for example, a predetermined change, or a change through trial and error to reduce the representative value of luminance included in the matching information. In the latter case, the shooting direction control unit 19 changes the shooting direction, for example, randomly or according to a predetermined rule. may be terminated. Also, the predetermined change may be to change the photographing direction in the horizontal plane by turning the manipulator 11 . The turning angle may be determined to be, for example, 60 degrees or 90 degrees.

If the compatibility information includes turning information, the photographing direction control unit 19 determines that the optical axis of the photographing unit 13 is the turning axis when the turning information indicates that the movement immediately after the moving mechanism 16 is turning. The shooting direction may be changed so as to approach the direction. Note that when the turning information does not indicate that the immediately following movement is turning, the imaging direction control unit 19 does not need to control the imaging direction according to the turning information. Here, if the turning information is flag-like information indicating whether or not the immediately following movement is turning, the photographing direction control unit 19 knows from the turning information whether or not the immediately following movement is turning. be able to. On the other hand, if the turning information is information indicating the speed of change in the direction of travel or the degree of change in the direction of travel, the photographing direction control unit 19 uses the turning information to determine whether the movement immediately after is a turn. It may be determined whether or not, and when turning, the photographing direction may be changed so that the optical axis of the photographing unit 13 approaches the direction of the turning axis. Specifically, when the speed of change in the direction of travel is greater than the threshold and the angle of change in the direction of travel is greater than the threshold, the imaging direction control unit 19 determines that the immediately following movement is a turn. good. Since the turning axis direction is usually the vertical direction, the imaging direction control unit 19 changes the imaging direction so that the imaging unit 13 can, for example, shoot upward or downward when the movement immediately after is turning. Alternatively, the photographing direction may be changed so as to photograph obliquely upward or obliquely downward. Above and below are above and below in the vertical direction. Therefore, when the photographing direction is changed so that the photographing unit 13 photographs upward, the optical axis of the photographing unit 13 becomes substantially vertical, and the ceiling or the like of the moving space may be photographed. In addition, since turning is normally performed in a horizontal plane, if the optical axis of the photographing unit 13 is substantially vertical during turning, the optical axis of the photographing unit 13 substantially coincides with the direction of the turning axis. . Therefore, when turning, only the angle of the captured image changes, and as a result, the tracking of the feature point becomes easier. Also, the change of the shooting direction according to the turning information may be performed when the shooting direction at that time is the horizontal direction. This is because when the imaging direction at that time is close to the vertical direction, there is no need to change the imaging direction even if the subsequent movement is a turn.

When ratio information is included in matching information, the shooting direction control unit 19 may change the shooting direction so that the ratio becomes larger when the ratio indicated by the ratio information is smaller than a predetermined threshold. Note that when the ratio indicated by the ratio information is larger than a predetermined threshold, the shooting direction control unit 19 does not have to control the shooting direction according to the ratio information. Further, when the ratio indicated by the ratio information is the same as the predetermined threshold value, the shooting direction control unit 19 may or may not control the shooting direction. The change in the imaging direction may be, for example, a predetermined change, or a trial-and-error change to increase the ratio indicated by the ratio information included in the matching information. In the latter case, the shooting direction control unit 19 changes the shooting direction, for example, randomly or according to a predetermined rule, and ends the shooting direction change when the ratio information becomes larger than the threshold as a result. may Also, the predetermined change may be to change the photographing direction in the horizontal plane by turning the manipulator 11 . The turning angle may be determined to be, for example, 60 degrees or 90 degrees. Also, the predetermined change may be to change the shooting direction so that the position of the shooting unit 13 is higher. If the object corresponding to the feature point included in the map cannot be photographed due to an obstacle, the object existing on the other side of the obstacle can be photographed by photographing from a position higher than the obstacle. Therefore, it is considered preferable to raise the photographing position. When changing the imaging direction so as to raise the position of the imaging unit 13, control is performed so that the azimuth direction of imaging is maintained, that is, the optical axis of the imaging unit 13 is moved in parallel. may By elevating the position of the photographing unit 13 so that the optical axis moves in parallel, for example, when the photographing unit 13 is photographing in the direction of travel, obstacles can be detected while photographing in the direction of travel is continued. It becomes possible to reduce the influence of

When the shooting direction is changed by the shooting direction control unit 19, the shooting angle and shooting position may be changed at a speed that can realize appropriate tracking. By doing so, it is possible to keep track of the self position when changing the shooting direction.

Further, after the shooting direction is changed according to the conformity information, the shooting direction may be returned to the original direction after a predetermined period of time has passed. For example, when the photographing direction is changed in accordance with the fact that the representative value of the brightness of the photographed image is large or the ratio indicated by the ratio information is small, the photographing direction control unit 19 controls the photographing after a predetermined period of time has elapsed. You can reverse the direction. Further, for example, when the imaging direction is changed in accordance with the fact that the immediately following movement is turning, the imaging direction control unit 19 indicates that the immediately following movement is not turning by means of the turning information included in the matching information. You may return to the original shooting direction after you become able to do so.

Next, the operation of the moving body 1 will be explained using the flowchart of FIG.
(Step S101) The current position acquisition unit 15 determines whether to acquire the current position. If the current position is to be acquired, the process proceeds to step S102; otherwise, the process proceeds to step S105. The current position acquisition unit 15 may periodically determine, for example, to acquire the current position.

(Step S102) The photographing unit 13 acquires a photographed image of the surroundings of the moving body 1 in the photographing direction at that time. Note that the acquired captured image may be stored in a recording medium (not shown).

(Step S103) The position estimating unit 14 extracts feature points from the acquired photographed image, and estimates the position of the photographing unit 13 using the extracted feature points and the feature points of the map. Also, the position estimation unit 14 may update the map using feature points extracted from the captured image. Note that, for example, position estimation may be performed each time a photographed image is acquired, and map updating may be performed each time the photographed image is acquired several times. Therefore, the position estimation unit 14 may update the map using the extracted feature points, for example, when it determines to update the map. The estimated position may be stored in a recording medium (not shown).

(Step S<b>104 ) The current position acquisition unit 15 acquires the current position of the moving body 1 using the angle of each axis of the manipulator 11 and the position of the imaging unit 13 acquired by the position estimation unit 14 . The acquired current position may be stored in a recording medium (not shown). Then, the process returns to step S101.

(Step S105) The acquisition unit 18 determines whether or not to acquire matching information. If the matching information is to be acquired, the process proceeds to step S106; otherwise, the process returns to step S101. The acquiring unit 18 may periodically determine, for example, to acquire matching information. It should be noted that the time interval for acquiring matching information is usually longer than the time interval for acquiring the current position.

(Step S106) The acquisition unit 18 acquires matching information. For example, the acquisition unit 18 may acquire the representative value of the luminance value using the captured image, and acquires the turning information by referring to the contents of future movement control managed by the movement control unit 17. Alternatively, the ratio information may be obtained by referring to the result of matching between the captured image and the map by the position estimation unit 14 . The acquired compatibility information may be stored in a recording medium (not shown).

(Step S107) The shooting direction control unit 19 uses the compatibility information acquired in step S106 to determine whether or not the shooting direction needs to be controlled. If it is necessary to control the photographing direction, the process proceeds to step S108. If not, the process returns to step S101. Note that, for example, when the matching information indicates that the representative value of the brightness of the captured image is large, or when the turning information indicates that the immediately following movement is turning, the ratio information indicates that the ratio of matching between the feature points of the captured image and the feature points of the map is low, it may be determined that the control of the capturing direction is necessary.

(Step S<b>108 ) The imaging direction control unit 19 changes the imaging direction of the imaging unit 13 by controlling the manipulator control unit 12 . Specifically, when it is indicated that the representative value of the brightness of the photographed image is large, the photographing direction control unit 19 rotates the manipulator 11 to change the photographing direction so that the representative value becomes small. You may Further, when the movement immediately after is turning, the photographing direction control unit 19 may change the photographing direction so that the photographing unit 13 photographs the upper part such as the ceiling. Further, when the ratio of matching between the feature points of the photographed image and the feature points of the map is low, the photographing direction control section 19 may change the photographing direction so as to raise the photographing position. Then, the process returns to step S101.

Although not included in the flowchart of FIG. 3, movement control using the current position acquired by the current position acquisition section 15 is performed by the movement control section 17 . Further, when the movement of the manipulator 11 to the work position is completed, the manipulator 11 is controlled by the manipulator control unit 12 so that predetermined work such as welding, painting, and assembly can be performed. good.

In addition, in the flowchart of FIG. 3, various information is included in the compatibility information, and after acquiring the compatibility information collectively, it is collectively determined whether or not to control the shooting direction according to the compatibility information. Although the case has been described, it does not have to be. Acquisition of information included in the matching information and determination of necessity of control of the imaging direction according to the acquired information may be repeatedly performed. In such a case, for example, first, turning information is acquired, and a determination is made as to whether or not control of the shooting direction is necessary. It is also possible to determine whether control of the shooting direction is necessary or not, and when control is not required, acquire ratio information and determine whether control of the shooting direction is necessary or not. It should be noted that if it is determined that the control is necessary in any of the determinations of whether or not control of the imaging direction is necessary, the control that is determined to be necessary may be performed and the process may return to step S101.

In addition, if the shooting direction is to be returned to the original direction after a predetermined period of time has elapsed after the shooting direction has been changed in accordance with the compatibility information, it is determined No in step S107, and the shooting direction is returned. The shooting direction control unit 19 may perform control to restore the shooting direction when the condition for is satisfied. Further, the order of processing in the flowchart of FIG. 3 is an example, and the order of each step may be changed as long as the same result can be obtained. In addition, in the flowchart of FIG. 3, the processing ends when the power is turned off or an interrupt for processing end occurs.

Next, the operation of the moving body 1 according to this embodiment will be described using a specific example. In this specific example, it is assumed that the moving object 1 moves in order of positions P1 to P6 along the route R2 shown in FIG. Also, arrows indicate shooting directions when the moving body 1 is moving among the positions P1 to P6. Also, in this specific example, it is assumed that the photographing is performed in the traveling direction except when the photographing direction is controlled according to the matching information.

First, when it is time to acquire the current position while the moving body 1 is moving at the position P1, shooting is performed in a shooting direction that matches the traveling direction indicated by the arrow, that is, upward in FIG. Using the captured image and the map, the position of the photographing unit 13 is estimated, and the current position of the moving body 1 is acquired using the position estimation result (steps S101 to S104). The current position is used for movement control by the movement control section 17 . It should be noted that after that, the photographing, the position estimation, and the acquisition of the current position will be repeated.

Further, when it comes time to acquire matching information, the acquiring unit 18 acquires a representative luminance value using the latest captured image at that time, and determines the content of future movement control managed by the movement control unit 17. The turning information is obtained by referring to it, and the ratio information is obtained using the result of matching between the latest photographed image and the map (steps S105 and S106). Note that the representative value of the luminance of the captured image acquired at the position P1 is smaller than a predetermined threshold value, the turning information indicates that the movement immediately after is not turning, and the ratio indicated by the ratio information is a predetermined value. Suppose it is larger than the threshold. Then, the photographing direction control unit 19 determines that the photographing direction control is unnecessary, and does not perform the photographing direction control (step S107). As a result, the photographing in the traveling direction is continued.

After that, when the moving body 1 reaches the position P2, the photographed image is overexposed due to the external light entering from the window on the left side of the traveling direction, and the luminance of the photographed image acquired at that time is representative of the brightness of the photographed image. Assume that the value is greater than a predetermined threshold (steps S105 and S106). Then, the imaging direction control unit 19 determines that control of the imaging direction is necessary, and rotates the imaging direction by 90 degrees via the manipulator control unit 12 (steps S107 and S108). Due to the turning, the photographing direction becomes rightward as shown in FIG. As a result, the external light incident on the photographing unit 13 changes from backlight to front light, and photographing can be performed with appropriate exposure. After a predetermined period of time has elapsed, the photographing direction control unit 19 returns the photographing direction to the standard setting of the traveling direction, as indicated by position P3 in FIG. Assuming that the range of the window is wider than that in FIG. 6 and the influence of external light is also present at position P3, it is determined that the representative value of the brightness of the captured image is large again after the shooting direction is returned to the standard setting. Accordingly, the photographing direction is changed to the right in the drawing.

When the moving body 1 reaches the position P4, the movement control unit 17 controls the moving mechanism 16 so that the moving body 1 turns 90 degrees. In this specific example, it is assumed that a change in the movement direction of 80 degrees or more is determined as turning. Then, by referring to the content of the movement control, the acquiring unit 18 acquires turning information indicating that the immediately following movement is turning (step S106). According to the turning information, the photographing direction control unit 19 determines that control of the photographing direction is necessary, and controls the manipulator control unit 12 so that the photographing unit 13 photographs vertically upward (steps S107 and S108). ). Therefore, the position estimator 14 can perform appropriate tracking even when the vehicle is turning, can appropriately acquire the turning angle, and can keep track of its own position. Note that when the turning is completed, turning information indicating that the movement immediately after is not turning is acquired. The manipulator control unit 12 is controlled so that photographing is performed.

After that, while the moving object 1 is moving at the position P5, it is assumed that a person M walks from the front. In this situation, the human M is present between the photographing unit 13 and the object corresponding to the feature points included in the map, and the ratio of matching between the feature points of the captured image and the feature points of the map is reduced. and Then, the acquiring unit 18 acquires ratio information indicating a decrease in the matching ratio (steps S105 and S106). If the ratio indicated by the ratio information is smaller than the predetermined threshold, the shooting direction control unit 19 determines that control of the shooting direction is necessary, and controls the manipulator control unit 12 so that the position of the shooting unit 13 is raised. Control. As a result, since the photographing is performed from a high place, the influence of the human being M is reduced, and the ratio of matching between the feature points of the photographed image and the feature points of the map can be increased. In this case as well, when the moving body 1 reaches the position P6 after the predetermined time has passed, the height of the photographing unit 13 is restored.

As described above, according to the mobile object 1 according to the present embodiment, it is possible to obtain a captured image that is more suitable for position estimation by controlling the shooting direction using matching information. As a result, the accuracy of the current position is improved, and more accurate movement can be realized. Moreover, since the representative value of the brightness of the photographed image is included in the matching information, it is possible to perform control so that the photographing is performed with a more appropriate exposure. In addition, since the turning information is included in the matching information, it is possible to perform photographing with appropriate tracking even when turning, and to keep track of the own position. In addition, by including the ratio information in the match information, it is possible to prevent the ratio of matching between the feature points of the photographed image and the feature points of the map from decreasing.

In the above-described embodiment, the case where the matching information includes the representative value of the luminance of the captured image, the turning information, and the ratio information has been mainly described, but this need not be the case. The matching information may include at least one piece of information out of the three pieces of information. Also, the matching information may include other information, and the imaging direction may be controlled using the information.

Further, in the above embodiments, each process or function may be implemented by centralized processing by a single device or single system, or may be implemented by distributed processing by multiple devices or multiple systems. It may be realized by

Further, in the above-described embodiment, when the information is passed between the components, for example, when the two components that exchange the information are physically different, one of the components output of information and reception of information by the other component, or one component if the two components that pass the information are physically the same from the phase of processing corresponding to the other component to the phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component Also, information such as thresholds, formulas, addresses, etc. used by each component in processing may be stored temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, each component or an accumulation section (not shown) may accumulate information in the recording medium (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

Further, in the above embodiment, if the information used in each component etc., for example, information such as thresholds, addresses and various set values used in processing by each component may be changed by the user, the above The user may or may not be able to change such information as appropriate, even if not explicitly stated in the description. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives a change instruction from the user and a change unit (not shown) that changes the information according to the change instruction. may The reception of the change instruction by the reception unit (not shown) may be, for example, reception from an input device, reception of information transmitted via a communication line, or reception of information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the mobile object 1 have communication devices, input devices, etc., the two or more components may physically have a single device. , or may have separate devices.

Further, in the above embodiments, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU. During the execution, the program execution unit may execute the program while accessing the storage unit or recording medium. Further, the program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). good. Also, this program may be used as a program constituting a program product. Also, the number of computers that execute the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible and are also included within the scope of the present invention.

As described above, according to the mobile object of the present invention, it is possible to acquire the current position with higher accuracy, and it is useful as a mobile object that performs position estimation using captured images.

1 moving body 11 manipulator 12 manipulator control unit 13 imaging unit 14 position estimation unit 15 current position acquisition unit 16 movement mechanism 17 movement control unit 18 acquisition unit 19 imaging direction control unit

Claims (4)

A mobile body that moves autonomously,
a manipulator;
a photographing unit mounted on the manipulator for acquiring a photographed image of the surroundings of the moving object;
a position estimating unit that estimates the position of the photographing unit using the photographed image and a map in the movement area of the moving object;
a current position acquiring unit that acquires the current position of the moving body using the position of the imaging unit estimated by the position estimating unit;
a moving mechanism for moving the moving body;
a movement control unit that controls the movement mechanism using the current position;
an acquisition unit that acquires matching information that is information indicating whether the captured image is suitable for position estimation and that includes a representative value of luminance of the captured image ;
a manipulator control unit that controls the manipulator;
When the representative value of the brightness of the photographed image is larger than a predetermined threshold, the representative value is reduced so that the photographed image suitable for position estimation is photographed using the matching information. and an imaging direction control unit that changes the imaging direction of the imaging unit by the manipulator control unit. A mobile body that moves autonomously,
a manipulator;
a photographing unit mounted on the manipulator for acquiring a photographed image of the surroundings of the moving object;
a position estimating unit that estimates the position of the photographing unit using the photographed image and a map in the movement area of the moving body;
a current position acquiring unit that acquires the current position of the moving body using the position of the imaging unit estimated by the position estimating unit;
a moving mechanism for moving the moving body;
a movement control unit that controls the movement mechanism using the current position;
an acquisition unit that acquires conformity information, which is information indicating whether the captured image is suitable for position estimation and includes turning information indicating whether the movement immediately after being performed by the moving mechanism is turning. ,
a manipulator control unit that controls the manipulator;
In order to capture a captured image suitable for estimating a position using the matching information, if the turning information indicates that the immediately following movement by the moving mechanism is turning, the imaging unit a photographing direction control unit for changing the photographing direction of the photographing unit by the manipulator control unit so that the optical axis of the photographing unit approaches the turning axis direction. A mobile body that moves autonomously,
a manipulator;
a photographing unit mounted on the manipulator for acquiring a photographed image of the surroundings of the moving object;
a position estimating unit that estimates the position of the photographing unit using the photographed image and a map in the movement area of the moving object;
a current position acquiring unit that acquires the current position of the moving body using the position of the imaging unit estimated by the position estimating unit;
a moving mechanism for moving the moving body;
a movement control unit that controls the movement mechanism using the current position;
Information indicating whether the captured image is suitable for position estimation, and includes ratio information indicating a matching ratio between the feature points extracted from the captured image and the feature points included in the map. an acquisition unit that acquires conformance information that is
a manipulator control unit that controls the manipulator;
If the ratio indicated by the ratio information is smaller than a predetermined threshold, the photographing is performed so that the ratio is larger than the predetermined threshold so that the photographed image suitable for position estimation is photographed using the matching information. and an imaging direction control unit that changes the imaging direction of the unit by the manipulator control unit . 4. The moving body according to claim 3 , wherein said shooting direction control unit changes the shooting direction so that the position of said shooting unit is higher when the ratio indicated by said ratio information is smaller than a predetermined threshold.

Images