US8831872B2 - Apparatus and method for estimating location of mobile body and generating map of mobile body environment using upper image of mobile body environment, and computer readable recording medium storing computer program controlling the apparatus - Google Patents
Apparatus and method for estimating location of mobile body and generating map of mobile body environment using upper image of mobile body environment, and computer readable recording medium storing computer program controlling the apparatus Download PDFInfo
- Publication number
- US8831872B2 US8831872B2 US11/331,058 US33105806A US8831872B2 US 8831872 B2 US8831872 B2 US 8831872B2 US 33105806 A US33105806 A US 33105806A US 8831872 B2 US8831872 B2 US 8831872B2
- Authority
- US
- United States
- Prior art keywords
- landmark
- unit
- image
- candidate
- mobile body
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
- G05D1/0253—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D3/00—Control of position or direction
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0268—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means
- G05D1/0272—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means comprising means for registering the travel distance, e.g. revolutions of wheels
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0268—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means
- G05D1/0274—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means using mapping information stored in a memory device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
Definitions
- the present invention relates to a mobile body such as a robot, and more particularly, to an apparatus and method for estimating the location of the mobile body and generating a map of the mobile body environment using an upper image of the mobile body environment, and a computer readable recording medium storing a computer program for controlling the apparatus.
- Another conventional method of recognizing a location of a robot and generating a map of a robot environment are also disclosed in the paper entitled “Real-Time 3D SLAM with Wide-Angle Vision”, Andrew J. Davison, Yolanda Gonzalez Cid, and Nobuyuki Kita, Proc. IFAC Symposium on Intelligent Autonomous Vehicles, 2004. All the conventional methods are affected by illumination.
- Another conventional method is disclosed in the paper entitled “Mobile Robot Localization and Mapping with Uncertainty Using Scale-Invariant Visual Landmarks”, Stephen Se, David Lowe, and Jim Little, International Journal of Robotics Research, Volume 21, Number 8, August 2002, pp. 735-758. In this conventional method, two cameras should be used and the method is affected by illumination variation so that a location of a robot and a map of a robot environment cannot be precisely respectively recognized and generated.
- An aspect of the present invention provides an apparatus for estimating a location of a mobile body and generating a map of a mobile body environment, the apparatus using an upper image of the mobile body environment obtained only by a simple low-priced camera.
- An aspect of the present invention also provides a method of estimating a location of a mobile body and generating a map of a mobile body environment, the method using an upper image of the mobile body environment obtained only by a simple low-priced camera.
- An aspect of the present invention also provides a computer readable recording medium storing a computer program for controlling the apparatus for estimating a location of a mobile body and generating a map of a mobile body environment.
- an apparatus for estimating a location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment including: a landmark generating unit observing corner points from the upper image obtained by photographing in an upper vertical direction in the mobile body environment and respectively generating landmarks from the observed corner points; and a location and map operation unit estimating the location of the mobile body and generating the map of the mobile body environment from the landmarks.
- a method of estimating a location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment including: observing corner points from the upper image obtained by photographing in an upper vertical direction in the mobile body environment and respectively generating landmarks from the observed corner points; and estimating the location of the mobile body and generating the map of the mobile body environment from the landmarks.
- a computer-readable storage medium encoded with processing instructions for causing a processor to execute a method of controlling an apparatus for estimating a location of a mobile body and generating a map of a mobile body environment using an upper image according to a process including: observing corner points from the upper image obtained by photographing in an upper vertical direction in the mobile body environment and respectively generating landmarks from the observed corner points; and estimating the location of the mobile body and generating the map of the mobile body environment from the landmarks.
- FIG. 1 is a block diagram of an apparatus for estimating a location of a mobile body and generating a map of a mobile body environment, the apparatus using an upper image of the mobile body environment, according to an embodiment of the present invention
- FIG. 2 is a flowchart illustrating a method of estimating a location of a mobile body and generating a map of a mobile body environment, the method using an upper image of the mobile body environment, according to another embodiment of the present invention
- FIG. 3 is a block diagram of an example of a landmark generating unit shown in FIG. 1 ;
- FIG. 4 is a flowchart illustrating an example of operation 20 shown in FIG. 2 ;
- FIG. 5 is a block diagram of an example of an image obtaining unit shown in FIG. 3 ;
- FIG. 6 is a flowchart illustrating an example of operation 50 shown in FIG. 4 ;
- FIG. 7A illustrates an example of an image inputted into a distortion calibrating unit
- FIG. 7B illustrates an example of an image outputted from the distortion calibrating unit
- FIG. 8 is a block diagram of an example of an image feature information extracting unit shown in FIG. 3 ;
- FIG. 9 is a flowchart illustrating an example of operation 54 shown in FIG. 4 ;
- FIGS. 10A and 10B are diagrams for understanding a corner point detecting unit
- FIG. 11 is a block diagram of an example of a landmark and landmark candidate predicting unit shown in FIG. 3 ;
- FIG. 12 is a flowchart illustrating an example of operation 56 shown in FIG. 4 ;
- FIG. 13 illustrates coordinates for explaining an operation of an image coordinate converting unit
- FIG. 14 is a diagram for understanding the selecting unit shown in FIG. 11 ;
- FIG. 15 is a diagram for understanding a location predicting unit
- FIG. 16 is a block diagram of an example of an observing and registering unit shown in FIG. 3 ;
- FIG. 17 is a flowchart illustrating an example of operation 58 shown in FIG. 4 ;
- FIG. 18 is an exemplary drawing for understanding the observing and registering unit shown in FIG. 16 ;
- FIG. 19 is a block diagram of an example of a landmark candidate processing unit shown in FIG. 3 ;
- FIG. 20 is a flowchart illustrating an example of operation 60 shown in FIG. 4 ;
- FIG. 21 is a drawing for understanding an operation of extracting height information using a trigonometrical relationship in the coordinate estimating unit shown in FIG. 19 ;
- FIG. 22 illustrates an example in which an image coordinate is converted into a mobile body coordinate
- FIG. 23 illustrates an example in which a mobile-body coordinate system is converted into a world coordinate system
- FIG. 24 is a diagram for explaining a processing model
- FIG. 25 is a diagram for explaining an observation model
- FIG. 26 shows the appearance of a mobile body to be tested
- FIGS. 27A and 27B are exemplary drawings illustrating a path on which the robot is to be moved
- FIG. 28 shows the appearance of the robot
- FIGS. 29A through 29D illustrate a figure for measuring the location of the robot
- FIG. 30 illustrates an example of measurement data when 45 sample data are obtained by measuring each of 15 points three times using the above-described measuring method
- FIG. 31 is a diagram for explaining a statistical analysis result of a distance error obtained in FIG. 30 , obtained using a data analysis program called Minitab;
- FIG. 32 is a diagram for explaining a statistical analysis result of an azimuth angle error, obtained using the data analysis program called Minitab.
- FIG. 1 is a block diagram of an apparatus for estimating location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment, according to an embodiment of the present invention.
- the apparatus of FIG. 1 comprises a landmark generating unit 10 , a location and map operation unit 12 , and an odometry processing unit 14 .
- FIG. 2 is a flowchart illustrating a method of estimating a location of a mobile body and generating a map of a mobile body environment using an upper image of the mobile body environment, according to another embodiment of the present invention.
- the method of FIG. 2 includes generating a landmark (operation 20 ) and estimating the location of the mobile body and generating the map of the mobile body (operation 22 ).
- the landmark generating unit 10 observes corner points of an upper image of the mobile body environment and generates landmarks corresponding to the observed corner points, respectively. In this case, the landmarks can be automatically generated.
- the landmark generating unit 10 searches for corner points that have been already used to generate landmarks or landmark candidates in a previous upper image, registers a corner point that has not been used to generate a landmark or a landmark candidate corresponding to a corner point in the previous upper image as a new landmark candidate, registers as a landmark a landmark candidate corresponding to a corner point that has been observed more than a predetermined number of times, and outputs the registered landmark to the location and map operation unit 12 using at least one of coordinates of corner points that have been observed in a current upper image and local feature descriptors (LFDs).
- LFDs local feature descriptors
- a corner point is a point that indicates a corner in the upper image
- the LFDs indicate the features of a corner point using the result of analyzing an ambient image of the corner point and classifies the corner points.
- the upper image is obtained by a camera (not shown) that is attached to the mobile body and faces upwards.
- a current upper image is an upper image that has been currently obtained
- a previous upper image is an image that was previously obtained and stored in an image buffer 32 shown in FIG. 3 , which will be described later.
- the upper image may be an image of a ceiling of an interior environment in which the mobile body moves or a wall surface adjacent to the ceiling. The mobile body can move while photographing such images.
- FIG. 3 is a block diagram of an example 10 A of the landmark generating unit 10 shown in FIG. 1 .
- the landmark generating unit 10 A of FIG. 3 includes an image obtaining unit 30 , an image buffer 32 , a landmark and landmark candidate predicting unit 34 , an image feature information extracting unit 36 , an observing and registering unit 38 , a landmark candidate processing unit 40 , and a landmark registering unit 42 .
- FIG. 4 is a flowchart illustrating an example 20 A of operation 20 .
- Operation 20 A of FIG. 4 includes obtaining and buffering an upper image (operations 50 and 52 ), extracting image feature information and predicting a previous landmark and a previous landmark candidate from a current upper image (operations 54 and 56 ), registering a new landmark candidate and managing the previous landmark and the previous landmark candidate (operation 58 ), and processing the landmark candidate and registering a corresponding landmark candidate as a landmark (operations 60 and 62 ).
- the image obtaining unit 30 of FIG. 3 obtains an image along an upper direction of the environment in which the mobile body moves, and outputs the obtained image to the image buffer 32 , the landmark and landmark candidate predicting unit 34 , and the image feature information extracting unit 36 , respectively.
- FIG. 5 is a block diagram of an example 30 A of the image obtaining unit 30 shown in FIG. 3 .
- Operation 30 A of FIG. 5 includes an upper image obtaining unit 70 , a distortion calibrating unit 72 , and a low pass filter (LPF) 74 .
- LPF low pass filter
- FIG. 6 is a flowchart illustrating an example 50 A of operation 50 shown in FIG. 4 .
- Operation 50 A of FIG. 6 includes obtaining an upper image (operation 80 ) and calibrating distortion of the upper image (operation 82 ), and performing low-pass-filtering (operation 84 ).
- the image obtaining unit 70 obtains the upper image in an upper vertical direction of the environment in which the mobile body is placed, and outputs the obtained upper image to the distortion calibrating unit 72 .
- the distortion calibrating unit 72 calibrates the distortion of the upper image obtained by the upper image obtaining unit 70 and outputs the distortion-calibrated result to the low pass filter (LPF) 74 .
- the distortion calibrating unit 72 may set a region of interest (ROI) 90 in the upper image obtained by the upper image obtaining unit 70 and calibrate a distortion only in the set ROI 90 .
- ROI region of interest
- FIG. 7A illustrates an example of an image inputted into the distortion calibrating unit 72
- FIG. 7B illustrates an example of an image outputted from the distortion calibrating unit 72 .
- the distortion calibrating unit 72 inputs the distorted image shown in FIG. 7A , calibrates the distortion of the image, and outputs the image shown in FIG. 7B as the distortion-calibrated result.
- the distortion calibrating unit 72 may calibrate the distortion only in the region of interest (ROI) 90 .
- the LPF 74 performs low-pass-filtering on the distortion-calibrated result so as to remove noise of harmonics from the distortion-calibrated result obtained by the distortion calibrating unit 72 and outputs the result of low-pass-filtering through an output terminal OUT 4 .
- the LPF 74 may be a Gaussian filter (not shown).
- the LPF 74 outputs the result of filtering shown in Equation 1 through an output terminal OUT 4 .
- I′(x,y) is a brightness intensity at a coordinate (x,y) of an image outputted from the LPF 74
- I(x+i,y+j) is a brightness intensity at a coordinate (x+i,y+j) of an image inputted into the LPF 74
- G [ G ⁇ ( 0 , 0 ) G ⁇ ( 0 , 1 ) G ⁇ ( 0 , 2 ) G ⁇ ( 1 , 0 ) G ⁇ ( 1 , 1 ) G ⁇ ( 1 , 2 ) G ⁇ ( 2 , 0 ) G ⁇ ( 2 , 1 ) G ⁇ ( 2 , 2 ) ] , ( 2 )
- G may be expressed as shown in Equation 3, for example.
- the image obtaining unit 30 may be implemented using only the upper image obtaining unit 70 .
- operation 50 may include only operation 80 .
- an image obtained by the upper image obtaining unit 70 is outputted to the image buffer 32 , the landmark and landmark candidate predicting unit 34 , and the image feature information extracting unit 36 , respectively.
- the image obtaining unit 30 may be implemented using only the upper image obtaining unit 70 and the distortion calibrating unit 72 .
- operation 50 includes only operations 80 and 82 .
- the distortion-calibrated result obtained by the distortion calibrating unit 72 is outputted to the image buffer 32 , the landmark and landmark candidate predicting unit 34 , and the image feature information extracting unit 36 , respectively.
- the image obtaining unit 30 may be implemented using only the upper image obtaining unit 70 and the LPF 74 .
- operation 50 includes only operations 80 and 84 .
- the LPF 74 performs low-pass-filtering on the upper image obtained by the upper image obtaining unit 70 and outputs the result of low-pass-filtering through an output terminal OUT 4 .
- the image buffer 32 buffers the image obtained by the image obtaining unit 30 and outputs the result of buffering to the landmark and landmark candidate predicting unit 34 as a previous upper image. That is, the image buffer 32 inputs a current upper image and simultaneously outputs a previous upper image.
- the image feature information extracting unit 36 extracts at least one of coordinates of corner points that have been observed at the current upper image obtained by the image obtaining unit 30 and local feature descriptors (LFDs) as image feature information and outputs the extracted image feature information to the observing and registering unit 38 .
- LFDs local feature descriptors
- FIG. 8 is a block diagram of an example 36 A of the image feature information extracting unit 36 shown in FIG. 3 .
- the image feature information extracting unit 36 A of FIG. 8 includes a corner point detecting unit 100 and a descriptor operation unit 102 .
- FIG. 9 is a flowchart illustrating an example 54 A of operation 54 shown in FIG. 4 .
- Operation 54 A of FIG. 9 includes detecting corner points (operation 110 ) and obtaining local feature descriptors (LFDs) (operation 112 ).
- LFDs local feature descriptors
- the corner point detecting unit 100 detects corner points from the current upper image inputted from the image obtaining unit 30 through an input terminal IN 2 and outputs coordinates of the detected corner points to the descriptor operation unit 102 as image feature information through an output terminal OUT 5 .
- FIGS. 10A and 10B are diagrams for understanding the corner point detecting unit 100 .
- FIG. 10A illustrates an example of an image inputted into the corner point detecting unit 100
- FIG. 10B illustrates cross-shaped corner points detected by the corner point detecting unit 100 .
- the corner point detecting unit 100 may detect corner points using the Harris corner point detection method, for example.
- the Harris corner point detection method is disclosed in the paper “A Combined Corner and Edge Detector”, C. J. Harris and M. Stephens, in Proc. 4 th Alvey Version Conf., Manchester, 1988, pp. 147-151.
- the corner point detecting unit 100 may input the image shown in FIG. 10A and detect corner points as shown in FIG. 10B .
- the corner point detecting unit 100 may determine a point where a value R expressed by Equation 4 is greater than 0 and which becomes a local maximum as a corner point.
- R det( M ) ⁇ k (tr( M )) 2 (4), where tr is a matrix trace, det is a matrix determinant, and M is a matrix with a gradient of image intensity obtained as Equation 5.
- k may be 0.04.
- I′ is the brightness intensity expressed by Equation 1 described above.
- the descriptor operation unit 102 analyzes images around the corner points detected by the corner point detecting unit 100 , performs an operation on a local feature descriptor (LFD) at each of the corner points using the analyzed result, and outputs the LFD as image feature information through an output terminal OUT 6 .
- the LFD on which an operation has been performed by the descriptor operation unit 102 may be a scale invariant feature transform (SIFT) descriptor.
- SIFT scale invariant feature transform
- the descriptor operation unit 102 inputs a current upper image from the image obtaining unit 30 through an input terminal IN 2 , inputs coordinates of the corner points detected by the corner point detecting unit 100 , and outputs an LFD comprised of 128 bytes, for example.
- the landmark and landmark candidate predicting unit 34 predicts whether a landmark and a landmark candidate in a previous upper image are located in a current upper image, and outputs the location of the landmark and the landmark candidate predicted from the current upper image to the observing and registering unit 38 .
- the observing and registering unit 28 of FIG. 3 can search for corner points that have been used to respectively generate landmarks and landmark candidates that have been already registered, from among the corner points detected from the current upper image.
- the track of the origin of a two-dimensional image coordinate system in each image and the location and azimuth angle of the mobile body when an image is inputted are stored, and the stored information is used to estimate a three-dimensional coordinate value of a landmark when the landmark candidate point is registered as a landmark.
- the landmark candidate predicting unit 34 of FIG. 3 inputs landmark information and the location and azimuth angle of the mobile body from the location and map operation unit 12 shown in FIG. 1 through an input terminal IN 1 , inputs landmark candidate information from the landmark registering unit 42 , inputs a previous upper image from the image buffer 32 , and inputs a current upper image from the image obtaining unit 30 .
- landmark information includes the location of a landmark, that is, a three-dimensional coordinate for the landmark in a world coordinate system and a local feature descriptor (LFD) for the landmark.
- LFD local feature descriptor
- the landmark candidate information includes the location of a landmark candidate, that is, a three-dimensional coordinate for the landmark candidate in the world coordinate system and an LFD for the landmark candidate.
- FIG. 11 is a block diagram of an example 34 A of the landmark and landmark candidate predicting unit 34 shown in FIG. 3 .
- the landmark and landmark candidate predicting unit 34 A of FIG. 11 includes an image coordinate converting unit 130 , a selecting unit 132 , and a location predicting unit 134 .
- FIG. 12 is a flowchart illustrating an example 56 A of operation 56 shown in FIG. 4 .
- Operation 56 A of FIG. 12 includes converting coordinates (operation 140 ), selecting a landmark and a landmark candidate that belong to a previous image, from landmarks and landmark candidates, and predicting a coordinate, that is, location of the landmark and the landmark candidate in a current upper image (operation 144 ).
- the image coordinate converting unit 130 inputs landmark and landmark candidate information that have been previously registered, based on a previous upper image through input terminals IN 3 , converts three-dimensional coordinates of locations of a landmark and a landmark candidate included in each of the inputted landmark and landmark candidate information into a two-dimensional coordinate based on an image coordinate system of the previous upper image using the location and azimuth angle of the mobile body when the previous upper image is inputted, and outputs landmark and landmark candidate information having the converted two-dimensional coordinates to the selecting unit 132 .
- the location and azimuth angle of the mobile body are inputted into the image coordinate converting unit 130 through an input terminal IN 4 .
- FIG. 13 illustrates coordinates for explaining an operation of the image coordinate converting unit 130 .
- the image coordinate system is defined by two coordinate axes X v and Y v (v denotes an image coordinate system)
- the mobile-body coordinate system is defined by two coordinate axes X r and Y r (r denotes a mobile-body coordinate system)
- the world coordinate system is defined by three coordinate axes X w , Y w , and Z w (w denotes a world coordinate system).
- the mobile body having two wheels 154 and 156 moves along a direction of arrow 152
- the location coordinate of the mobile body is expressed by (x,y, ⁇ ) w in the world coordinate system.
- reference numeral 150 of FIG. 13 denotes the image obtained by the image obtaining unit 30 in a physical space in which the mobile body moves, in consideration of a field of view of a camera.
- Locations of a landmark and a landmark candidate included in each of the landmark and landmark information inputted through input terminals IN 3 of the image coordinate converting unit 130 may be expressed in the world coordinate system as (x,y,z) w .
- each of the landmark and landmark candidates converted by the image coordinate converting unit 130 may be expressed in the image coordinate system (x,y) v .
- the selecting unit 132 selects only the landmark and the landmark candidate that belong to the previous upper image among the landmark and landmark candidates having the converted two-dimensional coordinates inputted from the image coordinate converting unit 130 and outputs the result of selection to the location predicting unit 134 .
- FIG. 14 is a diagram which illustrates a plurality of landmarks 160 , 162 , 164 , 166 , 168 , and 170 and a plurality of landmark candidates 180 , 182 , 184 , and 186 , for a better understanding of the selecting unit 132 shown in FIG. 11 .
- landmarks and landmark candidates of FIG. 14 are identified by symbols but the present invention is not limited to this.
- the selecting unit 132 selects only the landmark 168 and the landmark candidates 184 and 186 that belong to a previous upper image 190 , from the landmarks 160 , 162 , 164 , 166 , 168 , and 170 and the landmark candidates 180 , 182 , 184 , and 186 having the converted two-dimensional coordinates, and outputs information about the selected landmark 168 and the landmark candidates 184 and 186 to the location predicting unit 134 .
- the location predicting unit 134 calculates from the result of selection inputted from the selection unit 132 the amount of variation between the current upper image inputted from the image obtaining unit 30 through an input terminal IN 5 and the previous upper image inputted from the image buffer 32 of FIG. 3 through an input terminal IN 6 , that is, from each selected landmark and each selected landmark candidate, predicts a coordinate of a landmark and a landmark candidate which may exist in the current upper image, using the calculated amount of variation, and outputs the result of prediction to the observing and registering unit 38 through an output terminal OUT 7 .
- each of the coordinates of the result selected by the selecting unit 132 and the result predicted by the location predicting unit 134 is a two-dimensional coordinate in the image coordinate system.
- FIG. 15 is a diagram for a better understanding of the location predicting unit 134 .
- FIG. 15 illustrates a mobile body 200 in a previous upper image 190 and the mobile body 200 in a current upper image 192 .
- the location predicting unit 134 of FIG. 11 calculates the amount of variation between the current upper image 192 and the previous upper image 190 from each of a landmark 168 and landmark candidates 184 and 186 , which are selected by the selecting unit 132 , predicts each coordinate of the landmark 168 and the landmark candidates 184 and 186 which may exist in the current upper image, using the calculated amount of variation, and outputs the result of prediction to the observing and registering unit 38 through an output terminal OUT 7 .
- the location predicting unit 134 may predict locations of a landmark and a landmark candidate which may exist in the current upper image using an optical flow tracker.
- Optical flow is a motion vector which can detect the amount of movement and the amount of rotation of an image using brightness gradient information between the current upper image and the previous upper image.
- a Lukas-Kanade optical flow tracker which is widely used is disclosed in the paper entitled “An Iterative Image Registration Technique with an Application to Stereo Vision”, Lucas, B. and Kanade, T, Proc. of 7th International Joint Conference on Artificial Intelligence (IJCAI), pp. 674-679.
- operations 54 and 56 may be performed simultaneously, or operation 56 may be performed earlier than operation 54 .
- the observing and registering unit 38 compares information about the landmark and the landmark candidate, which are predicted by the landmark and landmark candidate predicting unit 34 , with image feature information inputted from the image feature information extracting unit 36 , observes and searches for corner points that have been used to generate landmarks or landmark candidates in the previous upper image, among corner points searched from the current upper image, using the compared result, registers a corner point that has not been used as a landmark or a landmark candidate in the previous upper image, as a new landmark candidate, and manages information about the landmark or the landmark candidate which correspond to the searched corner point.
- the observing and registering unit 38 outputs landmark candidate information about the new registered landmark candidate and the managed landmark candidate to the landmark candidate processing unit 40 and outputs landmark information about the managed landmark to the location and map operation unit 12 shown in FIG. 1 through an output terminal OUT 2 .
- FIG. 16 is a block diagram of an example 38 A of the observing and registering unit 38 shown in FIG. 3 .
- the observing and registering unit 38 A of FIG. 16 includes a landmark and landmark candidate searching unit 220 , first and second distance calculating units 222 and 226 , first and second comparing units 224 and 228 , a landmark and landmark candidate identifying unit 230 , a counter 232 , and a landmark candidate initializing unit 234 .
- FIG. 17 is a flowchart illustrating an example 58 A of operation 58 shown in FIG. 4 .
- Operation 58 A of FIG. 17 includes searching for the nearest predicted landmarks or landmark candidates corresponding to each of the corner points (operation 250 ) and registering a new landmark candidate according to a distance between the new landmark candidate and a corresponding corner point or managing a landmark and a landmark candidate (operations 252 to 262 ).
- FIG. 18 is an exemplary drawing for understanding the observing and registering unit 38 shown in FIG. 16 .
- observing and registering unit 38 of FIG. 18 includes a landmark 300 predicted by the landmark and landmark candidate predicting unit 34 from a current upper image, landmark candidates 306 and 312 predicted by the landmark and landmark candidate predicting unit 34 from the current upper image, and corner points 302 , 304 , 308 , 310 , and 314 searched from the current upper image inputted from the image feature information extracting unit 36 .
- the landmark and landmark candidate searching unit 220 inputs corner points included in image feature information inputted from the image feature information extracting unit 36 through an input terminal IN 7 and information about a landmark and/or a landmark candidate predicted by the landmark and landmark candidate predicting unit 34 through an input terminal IN 8 and searches for the nearest landmark or landmark candidate to the corner points among the predicted landmarks or landmark candidates.
- the landmark and landmark candidate searching unit 220 searches for the nearest landmark or landmark candidate to the corner point 302 , 304 , 308 , 310 or 314 . That is, the landmark and landmark candidate searching unit 220 searches for the predicted and nearest landmark 300 to the corner point 302 , the predicted and nearest landmark candidate 306 to the corner point 304 , the predicted and nearest landmark candidate 312 to the corner point 308 , the predicted and nearest landmark candidate 312 to the corner point 310 , and the predicted and nearest landmark candidate 312 to the corner point 314 . In this case, the landmark and landmark candidate searching unit 220 forms pairs by binding the predicted and nearest landmarks or landmark candidates to the respective corner points and outputs information about each pair as shown in FIG. 18 .
- the corner point 302 and the predicted landmark 300 are bound as a pair 270
- the corner point 304 and the predicted landmark candidate 306 are bound as a pair 278
- the corner point 308 and the predicted landmark candidate 312 are bound as a pair 272
- the corner point 310 and the predicted landmark candidate 312 are bound as a pair 274
- the corner point 314 and the predicted landmark candidate 312 are bound as a pair 276 . That is, elements that belong to each pair 270 , 272 , 274 , 276 or 278 are a corner point and a predicted landmark or landmark candidate.
- the first distance calculating unit 222 calculates a distance between the corner point and the searched landmark or landmark candidate and outputs the calculated distance to the first comparing unit 224 .
- the first comparing unit 224 compares the distance calculated by the first distance calculating unit 222 with the first predetermined distance and outputs the compared result.
- the first predetermined distance may be a 15-pixel distance.
- a 1-pixel distance means a spatial length corresponding to one pixel length.
- the first distance calculating unit 222 calculates a distance between the corner point and the searched landmark or landmark candidate and outputs each calculated distance to the first comparing unit 224 , with regard to each of the pairs 270 , 272 , 274 , 276 , and 278 inputted from the landmark and landmark candidate searching unit 220 .
- a distance between a local feature descriptor (LFD) of a corner point and an LFD of the searched landmark or landmark candidate is equal to or smaller than a second predetermined distance.
- the second distance calculating unit 226 calculates an Euclidean distance between the LFD of the corner point and the LFD of the searched landmark or landmark candidate and outputs the calculated distance to the second comparing unit 228 .
- the second distance calculating unit 226 inputs the LFDs from the landmark and landmark candidate searching unit 220 .
- the second comparing unit 228 compares the distance calculated by the second distance calculating unit 226 with a second predetermined distance and outputs a compared result.
- the second predetermined distance may be 638.
- the second distance calculating unit 226 selects each pair from the pairs 270 , 272 , 274 , 276 , and 278 inputted from the landmark and landmark candidate searching unit 220 for which it is recognized through the result compared by the first comparing unit 224 that the distance between the corner point and the searched landmark or landmark candidate is equal to or smaller than the first predetermined distance, calculates an Euclidian distance between the LFD of the corner point that belongs to the selected pair and the LFD of the searched landmark or landmark candidate that belongs to the selected pair and outputs the calculated distance to the second comparing unit 228 .
- the landmark and landmark candidate identifying unit 230 identifies whether a landmark or a landmark candidate exists at the nearest location to the corner point and outputs the identified result to the counter 232 .
- the landmark and landmark candidate identifying unit 230 identifies whether a landmark or a landmark candidate exists at the nearest location to the corner point and outputs an identified result to the counter 232 , with respect to each pair for which it is recognized through the result compared by the second comparing unit 228 that the distance between the LFDs is equal to or smaller than the second predetermined distance, from pairs for which the distance between the corner point inputted from the landmark and landmark candidate searching unit 220 and the landmark or landmark candidate is equal to or smaller than the first predetermined distance. For example, when it is recognized for the pair 270 of FIG.
- the landmark and landmark candidate identifying unit 230 identifies a landmark or a landmark candidate that forms a pair with the corner point 302 as the landmark 300 .
- the landmark and landmark candidate identifying unit 230 identifies a landmark or a landmark candidate that forms a pair with the corner point 310 as the landmark candidate 312 .
- the counter 232 counts a landmark flag that indicates how many times a landmark has been observed, in an ascending series, and outputs the counted result through an output terminal OUT 8 .
- the counter 232 counts a landmark candidate flag that indicates how many times a landmark candidate has been observed, in an ascending series and outputs the counted result through an output terminal OUT 8 .
- the counter 232 counts a landmark flag that indicates how many times a landmark has been observed, in an ascending series or counts a landmark candidate flag that indicates the number in which a landmark candidate has been observed, in an ascending series, in response to the result identified by the landmark and landmark candidate identifying unit 230 .
- the result for which the landmark flag is counted by the counter 232 in an ascending series is outputted as landmark information
- the result in which the landmark candidate flag is counted by the counter 232 in an ascending series is outputted as landmark candidate information.
- the counter 232 counts a landmark flag value included in landmark information in an ascending series, and since two landmark candidates 306 and 312 have been observed, the counter 232 counts each landmark candidate flag value included in landmark candidate information in an ascending series.
- the corner point more distant from the searched landmark or landmark candidate than the first predetermined distance or the corner point having an LFD more distant from an LFD of the searched landmark or landmark candidate than the second predetermined distance is determined as a new landmark candidate and information about the determined new landmark candidate is registered.
- the landmark candidate initializing unit 234 determines the corner point included in each pair for which it is recognized through the result compared by the first comparing unit 222 that the distance between the corner point and the searched landmark or landmark candidate is greater than the first predetermined distance, as a new landmark candidate and registers information about the determined new landmark candidate as landmark candidate information.
- the landmark candidate initializing unit 234 determines the corner point included in each pair for which it is recognized through the result compared by the second comparing unit 226 that the distance between the LFDs is greater than the second predetermined distance as a new landmark candidate and registers information about the determined new landmark candidate as landmark candidate information.
- a landmark candidate flag included in landmark candidate information registered as the new landmark candidate is initialized to 0.
- the corner points 308 , 314 , and 304 respectively included in the pairs 272 , 276 , and 278 shown in FIG. 18 have not been used to generate both landmarks and landmark candidates in a previous upper image
- the corner points 308 , 304 , and 314 are determined as a new landmark candidate and landmark candidate information about the determined landmark candidates 304 , 308 , and 314 is registered. For example, if it is recognized that the distance between the corner point 308 and the landmark candidate 312 included in the pair 272 shown in FIG.
- the corner point 308 is determined as a new landmark candidate, and the landmark candidate information about the determined landmark candidate 308 is registered.
- the landmark candidate processing unit 40 removes overlapping landmark candidates from the landmark candidates registered by the observing and registering unit 38 , estimates a three-dimensional image coordinate from a two-dimensional image coordinate of non-overlapping landmark candidates, and outputs the non-overlapping landmark candidates having the estimated three-dimensional image coordinate to the landmark registering unit 42 .
- each coordinate of the registered landmark candidates inputted into the landmark candidate processing unit 40 from the observing and registering unit 38 is a two-dimensional coordinate expressed in the image coordinate system.
- FIG. 19 is a block diagram of an example 40 A of the landmark candidate processing unit 40 shown in FIG. 3 .
- the landmark candidate processing unit 40 A of FIG. 19 includes a third distance calculating unit 330 , a third comparing unit 332 , a landmark candidate removing unit 334 , and a coordinate estimating unit 336 .
- FIG. 20 is a flowchart illustrating an example 60 A of operation 60 shown in FIG. 4 .
- Operation 60 A of FIG. 20 includes removing an overlapping landmark candidate or estimating a three-dimensional coordinate of an non-overlapping landmark candidate according to the result of comparing a minimum distance with a third predetermined distance (respective operations 350 through 356 ).
- the third distance calculating unit 330 calculates a minimum distance between one registered landmark candidate and another registered landmark candidate that is the nearest to the registered landmark candidate and outputs the calculated minimum distance to the third comparing unit 332 , from each of the registered landmark candidates inputted through an input terminal IN 9 from the observing and registering unit 38 .
- the third comparing unit 332 compares each minimum distance inputted from the third distance calculating unit 330 with the third predetermined distance and outputs compared results to the landmark candidate removing unit 334 .
- the third predetermined distance may be set to a 5-pixel distance.
- the corner points 308 , 304 , and 314 shown in FIG. 18 are determined as new landmark candidates and landmark candidate information about the determined landmark candidates 308 , 304 , and 314 is registered by the observing and registering unit 38 , if a minimum distance between the corner points 308 and 304 is equal to or smaller than the third predetermined distance, one of the new landmark candidates 308 and 304 is removed.
- the coordinate estimating unit 336 estimates three-dimensional image coordinates of non-overlapping landmark candidates inputted through an input terminal IN 9 from the observing and registering unit 38 and outputs the landmark candidate having the estimated three-dimensional image coordinate to the landmark registering unit 42 through an output terminal OUT 10 .
- the coordinate estimating unit 336 inputs the landmark candidates that have not been removed from the landmark candidate removing unit 334 , from the landmark candidate removing unit 334 , estimates a three-dimensional image coordinates of the inputted and non-overlapping landmark candidates, and outputs the landmark candidate having the estimated three-dimensional coordinates to the landmark registering unit 42 through an output terminal OUT 10 .
- the coordinate estimating unit 336 shown in FIG. 19 can more easily estimate three-dimensional image coordinates from a two-dimensional coordinate track of the non-overlapping landmark candidates, and a location and an azimuth angle of the mobile body.
- FIG. 21 illustrates coordinates of locations P 1 , P 2 , and P L in a world coordinate system for a better understanding of height information used in the coordinate estimating unit 336 shown in FIG. 19 .
- the location and azimuth angle of the mobile body at location P 1 are expressed as (x R1 , y R1 , ⁇ R1 ) w
- the location and azimuth angle of the mobile body at location P 2 are expressed as (x R2 , y R2 , ⁇ R2 ) w
- the landmark P L is placed in (x L , y L , ⁇ L ) w on a world coordinate system and it is observed using the upper image at each of location P 1 and P 2 , (x L1 , y L1 ) v and (x L2 , y L2 ) v can be obtained in each image coordinate system.
- the relationship between the image coordinate system and the world coordinate system is shown in FIG. 13 .
- the landmark in the world coordinate system is expressed as the same point (x L , y L , ⁇ L )w
- the coordinate of the landmark at P 1 in the image coordinate system is expressed as (x L1 , y L1 )v
- the coordinate of the landmark at P 2 in the image coordinate system is expressed as (x L2 , y L2 )v.
- a point in the image coordinate system is a point in which the landmark P L is projected onto a plane photographed by a robot camera.
- the coordinate estimating unit 336 can estimate the height of an actual landmark candidate placed at the location P L using a trigonometrical relationship from the distance between the location P 1 and the location P 2 of the mobile body and the angles ⁇ R1 and ⁇ R2 and values (x L1 , y L1 ) v and (x L2 , y L2 ) v on a landmark obtained from each image coordinate system.
- FIG. 22 illustrates an example of changing image coordinates from the image coordinate system into the mobile-body coordinate system
- FIG. 23 illustrates an example of changing mobile body coordinates from the mobile-body coordinate system into the world coordinate system.
- the coordinate estimating unit 336 converts two-dimensional coordinates of a landmark candidate in the image coordinate system into the mobile-body coordinate systems, as shown in Equation 6.
- [ x v y v 1 ] R is a two-dimensional coordinate of the landmark candidate expressed in the mobile-body coordinate system
- V is a two-dimensional coordinate of a landmark candidate expressed in the image coordinate system
- o x and o y are optical center coordinates in a vision coordinate system
- s x and s y are scale factors in each direction in the vision coordinate system.
- Equation 7 [ 0 - s y s y ⁇ o y s x 0 - s x ⁇ o x 0 0 1 ] may be expressed as a transform matrix T V R , as shown in Equation 7.
- the coordinate estimating unit 336 converts the two-dimensional coordinate of the landmark candidate in the mobile-body coordinate system into the world coordinate system, as shown in Equation 8.
- [ x 1 y 1 1 ] R is a two-dimensional coordinate of a landmark candidate expressed in the mobile-body coordinate system
- [ x 1 y 1 1 ] W is a two-dimensional coordinate of a landmark candidate expressed in the world coordinate system, and T W R may be expressed as a transform matrix as shown in Equation 9.
- the coordinate estimating unit 336 can obtain a three-dimensional coordinate in the world coordinate system of the landmark candidate as shown in Equation 10.
- X L ( H T H ) ⁇ 1 H T X R (10), where X L is a three-dimensional coordinate in the world coordinate system of the landmark candidate and expressed as shown in Equation 11, X R is a two-dimensional coordinate in the image coordinate system of the landmark candidate and expressed as shown in Equation 12, T is a matrix transpose, and H is expressed as in Equation 13 and may be induced from Equation 14 that shows the coordinate conversion relationship and Equation 15 that shows a trigonometrical relationship.
- X L [ x 1 y 1 z 1 ] ( 11 )
- X R [ x R ⁇ ⁇ 1 y R ⁇ ⁇ 1 ⁇ x RN y RN ] ( 12 )
- H [ 10 - ( x 11 ′ - x R ⁇ ⁇ 1 ) f 10 - ( x 1 ⁇ N ′ - x R ⁇ ⁇ N ) f ⁇ 01 - ( y 11 ′ - y R ⁇ ⁇ 1 ) f 01 - ( y 1 ⁇ N ′ - y R ⁇ ⁇ N ) f ] ( 13 ) [ x 11 y 11 ] V -> [ x 11 ′ y 11 ′ ] W ⁇ [ x 1 ⁇ N y 1 ⁇ N ] V -> [ x 1 ⁇ N ′ y 1 ⁇ N ′ ] W ( 14 ) z 1 f ⁇ [ x 11 ′ - x
- the landmark generating unit 10 shown in FIG. 3 may not include the landmark candidate processing unit 40 .
- operation 20 shown in FIG. 4 may not include operation 60 .
- the landmark registering unit 42 registers the landmark candidate that has been observed as a corner point more than a predetermined number of times among the landmark candidates as a landmark, outputs the registered landmark through an output terminal OUT 3 , and outputs landmark candidate information about the landmark candidate observed as the corner point less than the predetermined number of times among the landmark candidates to the landmark and landmark candidate predicting unit 34 .
- the predetermined number may be set to 5, for example.
- the landmark registering unit 42 registers a new landmark by increasing the size of a covariance matrix of an extended Kalman filter (EKF), inserting a related-landmark location into a submatrix of the covariance matrix, and inserting related-landmark noise into a noise covariance matrix.
- EKF extended Kalman filter
- the location and map operation unit 12 estimates the location of the mobile body using landmark information inputted from the landmark generating unit 10 and, simultaneously, generates an environment of a map of the mobile body, outputs the estimated location and the generated map through an output terminal OUT 1 , and outputs the landmark information and the location and the azimuth angle of the mobile body to the landmark generating unit 10 , that is, to the landmark and landmark candidate predicting unit 34 shown in FIG. 3 .
- the apparatus for estimating the location and generating the environment map of the mobile body shown in FIG. 1 may further include an odometry processing unit 14 .
- the odometry processing unit 14 controls an odometry of the mobile body and outputs information about the odometry to the location and map operation unit 12 .
- the information about the odometry may be a sensor value obtained from encoders (not shown) respectively attached to right and left wheels of the mobile body.
- the location and map operation unit 12 estimates the location of the mobile body from the information about the odometry inputted from the odometry processing unit 14 and landmarks inputted from the landmark generating unit 10 and simultaneously, generates an environment map of the mobile body, and outputs information about the landmark to the landmark generating unit 10 .
- the location and map operation unit 12 may be implemented by a simultaneous localization and map building (SLAM) operation unit (not shown) which recognizes the location of the mobile body using the landmarks and, simultaneously, generates a map of the landmarks using an EKF.
- SLAM simultaneous localization and map building
- EKF-based SLAM includes a theoretical proof about a general methodology and an SLAM technique but does not include a series of methods, for example, a motion model and an observation model proposed by the present invention. That is, the performance of the location of the mobile body and azimuth angle operation and map generation is improved using the landmark generating unit 10 shown in FIG. 1 , compared to a general EKF-based SLAM technique.
- FIG. 24 is a diagram for explaining a processing model, which illustrates a coordinate (x k ,y k ) of a mobile body 400 in a world coordinate system.
- a vehicle processing model that is, a location model of the mobile body 400 is expressed as shown in Equation 16
- a landmark processing model that is, a model of a map may be expressed as shown in Equation 17.
- FIG. 25 is a diagram for explaining an observation model.
- x, y, and z are the axes of the world coordinate system
- x v and y v are the axes of the image coordinate system
- (x j , y 1 , z 1 ) is a coordinate of a landmark in the world coordinate system
- (x R , y R and ⁇ R ) is a coordinate of the landmark in the mobile-body coordinate system.
- a distance d v between central points (x v , y v , z v ) and a point 402 on the image coordinate system may be expressed as shown in Equation 18, and an angle ⁇ v formed by vector Z between the axis x v and the point 402 in the image coordinate system may be expressed as shown in Equation 19.
- d v ( x R - x L ) 2 + ( y R - y l ) 2 ⁇ f z l , ( 18 ) where f is a scale factor representing a focal length of a camera lens.
- the above-described embodiments of the present invention can be written as a program in a computer readable recording medium and executed in a computer.
- Examples of the computer readable recording medium include magnetic storage media (such as read-only memory (ROM), floppy disks or hard disks), optical readable media (such as CD ROMs or DVDs etc.), and carrier waves (such as data transmission through the Internet).
- corner points in an upper image obtained by taking a picture in an upper vertical direction in the environment in which a mobile body is placed are observed, landmarks are generated from the observed corner points, the location of the mobile body is estimated from the landmarks, and an environment map of the mobile body is generated.
- corner points that have been used to generate landmarks or landmark candidates in a previous upper image are observed and searched, a corner point that has not been used as a landmark or a landmark candidate in the previous upper image is registered as a new landmark candidate, and landmark candidates that have been observed as a corner point more than a predetermined number of times among landmark candidates, are registered as a landmark, using at least one of coordinates of corner points that indicate corner in the current upper image and LFDs that classify corner points.
- registering of the landmark includes obtaining an image in an upper vertical direction of the environment in which the mobile body is placed, buffering the obtained image and determining the buffered result as a previous upper image, extracting at least one of coordinates of the corner points of the current upper image and LFDs as image feature information, predicting a landmark and a landmark candidate in the previous upper image from the current upper image, comparing information about the predicted landmark and landmark candidate with image feature information, observing corner points that have been used to generate landmarks or landmark candidates in the previous upper image among the corner points using the compared result, registering a corner point that has not been used to generate a landmark or a landmark candidate in the previous upper image as a new landmark candidate, and registering landmark candidates that have been observed as a corner point more than a predetermined number of times among the landmark candidates, as a landmark.
- the obtaining of the image includes obtaining an image in an upper vertical direction of the environment in which the mobile body is placed and correcting distortion of the obtained upper image.
- the obtaining of the image further includes low-pass-filtering the distortion-calibrated result.
- the obtaining of the image includes obtaining an image in the upper vertical direction of the environment in which the mobile body is placed and low-pass-filtering the obtained upper image.
- the extracting of the image feature information includes detecting corner points from the current upper image, obtaining coordinates of the detected corner points, analyzing images around the detected corner points and obtaining an LFD at each of the corner points using the analyzed result.
- the predicting of the locations of the landmark and landmark candidate from the current upper image includes converting three-dimensional coordinates of the locations of the registered landmark and landmark candidate in the previous upper image using the location and azimuth angle of the mobile body when the previous upper image is inputted based on the image coordinate system of the previous upper image, selecting only a landmark and a landmark candidate that belong to the previous upper image among the landmarks and landmark candidates having the converted two-dimensional coordinate, and calculating the amount of variation between the current upper image and the previous upper image from the selected result and predicting a landmark and a landmark candidate that may exist in the current upper image, using the calculated amount of variation.
- the observing of the corner points and registering of the new landmark candidate includes searching for the predicted and the nearest landmark or landmark candidate to the corner point with respect to each of the corner points, determining whether the distance between the corner point and the searched landmark or landmark candidate is equal to or smaller than the first predetermined distance, with respect to each of the corner points, determining whether the distance between the LFD of the corner point and the LFD of the searched landmark or landmark candidate is equal to or smaller than the second predetermined distance, with respect to each of the corner points in which the distance between the searched landmark or landmark candidate is equal to or smaller than the first predetermined distance, if it is determined that the distance between the LFD of the corner point and the LFD of the searched landmark or landmark candidate is equal to or smaller than the second predetermined distance, determining whether the landmarks or landmark candidates exist at the nearest location to the corner point, if it is determined that the landmark exists at the nearest location to the corner point, counting a landmark flag that indicates how many times a landmark has been observed, in an ascending series, if it is determined that the landmark candidate exists at
- the registering of the landmark further includes removing overlapping landmark candidates from the registered landmark candidates and estimating a three-dimensional image coordinate from the two-dimensional image coordinate of the non-overlapping landmark candidates.
- the estimating of the image coordinate includes obtaining a minimum distance between the nearest landmark candidates with respect to the registered landmark candidates, determining whether the minimum distance is equal to or smaller than the third predetermined distance, determining the landmark candidates having the minimum distance equal to smaller than the third predetermined distance to be overlapped with one another and removing the overlapping landmark candidates, determining the landmark candidates having the minimum distance greater than the third predetermined distance not to be overlapped with one another, and estimating a three-dimensional image coordinate of the non-overlapping landmark candidates.
- the result obtained by dividing the sum of distance errors at each location of the mobile body into a total number of samples is defined as a distance mean error
- the result obtained by dividing the sum of azimuth angle errors at each location of the mobile body into a total number of samples is defined as an azimuth angle mean error.
- a distance error e d may be expressed in cm as shown in Equation 20
- an azimuth angle error e ⁇ may be expressed in degrees as shown in Equation 21.
- e d ⁇ square root over (( x d ⁇ x ) 2 +( y d ⁇ y ) 2 ) ⁇ square root over (( x d ⁇ x ) 2 +( y d ⁇ y ) 2 ) ⁇ (20)
- e ⁇
- the mobile body When the environment in which the mobile body moves is a living room of an apartment, a path of the living room on which the mobile body is to move is determined previously, the mobile body automatically moves along the path, the map of the living room is generated in real-time and the location of the mobile body is recognized.
- the mobile body moves to a predetermined number of points on the path, for example, 15 points, and then, the location and azimuth angle of the mobile body are measured a predetermined number of times at each point, for example, three times.
- a distance error and an azimuth angle error between the calculated locations and azimuth angles and the measured locations and azimuth angles are obtained as a predetermined number of samples, for example, 45 samples.
- FIG. 26 shows the appearance of a mobile body to be tested, which illustrates a wheel driving unit 550 , a robot 551 , an upper image camera 552 , and a ceiling 554 .
- the image feature information extracting unit 36 shown in FIG. 3 can detect a plurality of corner points 560 to 578 from the ceiling 554 .
- the wheel driving unit 550 shown in FIG. 26 performs the function of the odometry processing unit 14 shown in FIG. 1 , and the upper image camera 552 may be embedded in the image obtaining unit 30 shown in FIG. 3 .
- FIGS. 27A and 27B are exemplary drawings illustrating a path on which the mobile body is to move.
- FIG. 27A is a plane view of an apartment including a living room of the apartment where the robot travels
- FIG. 27B is an exemplary drawing of an odometry path of the robot.
- the dimensions of the apartment are shown in mm.
- the path shown in FIG. 27B is in the world coordinate system.
- the robot automatically learns the map along the path in the directions of the arrows shown in FIG. 27B .
- the robot starts learning from learning starting points ( 0 , 0 )( 600 ) shown in FIG. 27B .
- the coordinate values of the robot is obtained through actual measurement from the world coordinate system at each of points 602 to 622 shown in FIG. 27B and after the learning is completed, the location and azimuth angle error are measured at the points.
- the locations and azimuth angles of the mobile body are measured using a ruler and a protractor in the following order.
- FIG. 28 shows an appearance of the robot 551
- FIGS. 29A through 29D illustrate operations of measuring of the location and azimuth angle of the robot 551 .
- FIG. 30 illustrates an example of measurement data when 45 sample data are obtained by measuring locations and azimuth angles of the mobile body at each of the 15 points three times using the above-described measuring method.
- FIG. 31 is a diagram for explaining a statistical analysis result of a distance error obtained in FIG. 30 , obtained using the data analysis program called Minitab when an upper specification limit (USL) is 10 cm.
- the vertical axis represents a probability distribution and the horizontal axis represents a range of a distance error.
- a mean distance error Mean with respect to 30 observation data number Sample N is 5.9018 cm and a sigma level used to measure the degree of distribution is 2.74 using Z.Bench+1.5 with respect to an overall capability.
- a sigma level is an index for indicating a defective rate, and a 6-signal level means the level at which 3.4 defects occurs per one million units.
- FIG. 32 is a diagram for explaining a statistical analysis result of an azimuth angle error, obtained using a data analysis program called Minitab when an upper specification limit (USL) is 5 degree.
- the vertical axis represents a probability distribution of observed azimuth angle data and the horizontal axis represents a range of an azimuth angle error.
- a mean distance error Mean with respect to 30 observation data number Sample N is 1.94333 degree and a sigma level used to measure the degree of distribution is 3.43 using Z.Bench+1.5 with respect to an overall capability.
- the apparatus and method for estimating the location of the mobile body and generating the map of the mobile body environment since a natural landmark that is robust to a change of the environment in which the mobile body moves, for example, illumination, etc., is automatically generated from the upper image obtained from the environment in which the mobile body moves, an artificial landmark such as a separate attachment, for example, infrared landmark, a color pattern, etc., estimation of the location of the mobile body and generation of the map of the mobile body environment can be more correctly performed compared to a conventional method.
- a local feature descriptor such as an SIFT descriptor that indicates the feature of a corner point
- LFD local feature descriptor
- a brightness gradient of an image is uniform such that a landmark that is robust to a change of illumination of the mobile body is searched for and registered and estimating the location of the mobile body and generating the map of the mobile body is correctly performed. That is, a mean distance error of the mobile body is maintained within about 6 cm and an azimuth angle mean error is maintained within about 2 degree.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Remote Sensing (AREA)
- Aviation & Aerospace Engineering (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Electromagnetism (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Instructional Devices (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
Description
where I′(x,y) is a brightness intensity at a coordinate (x,y) of an image outputted from the
where G may be expressed as shown in
R=det(M)−k(tr(M))2 (4),
where tr is a matrix trace, det is a matrix determinant, and M is a matrix with a gradient of image intensity obtained as
where I′ is the brightness intensity expressed by
where
is a two-dimensional coordinate of the landmark candidate expressed in the mobile-body coordinate system, and
is a two-dimensional coordinate of a landmark candidate expressed in the image coordinate system, ox and oy are optical center coordinates in a vision coordinate system, and sx and sy are scale factors in each direction in the vision coordinate system.
may be expressed as a transform matrix TV R, as shown in Equation 7.
where
is a two-dimensional coordinate of a landmark candidate expressed in the mobile-body coordinate system,
is a two-dimensional coordinate of a landmark candidate expressed in the world coordinate system, and TW R may be expressed as a transform matrix as shown in Equation 9.
X L=(H T H)−1 H T X R (10),
where XL is a three-dimensional coordinate in the world coordinate system of the landmark candidate and expressed as shown in Equation 11, XR is a two-dimensional coordinate in the image coordinate system of the landmark candidate and expressed as shown in
x k+1 =x k +T ν cos(θ k)
y k+1 =y k +T ν sin(θ k)
θk+1=θk +T w (16),
where (xk,yk,θk) is the location and angle of the mobile body in a previous upper image in a world coordinate system, (xk+1,yk+1,θk+1) is the location and angle of the mobile body in a current upper image in the world coordinate system, v is the linear velocity of the mobile body, w is the angular velocity of the mobile body, and T is a sampling time.
where f is a scale factor representing a focal length of a camera lens.
e d=√{square root over ((x d −x)2+(y d −y)2)}{square root over ((x d −x)2+(y d −y)2)} (20)
e θ=|θd−θ| (21)
Claims (19)
R=det(M)−k(tr(M))2
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR10-2005-0006573 | 2005-01-25 | ||
KR1020050006573A KR100776215B1 (en) | 2005-01-25 | 2005-01-25 | Apparatus and method for estimating location and generating map of mobile body, using upper image, computer-readable recording media storing computer program controlling the apparatus |
Publications (2)
Publication Number | Publication Date |
---|---|
US20060165276A1 US20060165276A1 (en) | 2006-07-27 |
US8831872B2 true US8831872B2 (en) | 2014-09-09 |
Family
ID=36216787
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/331,058 Expired - Fee Related US8831872B2 (en) | 2005-01-25 | 2006-01-13 | Apparatus and method for estimating location of mobile body and generating map of mobile body environment using upper image of mobile body environment, and computer readable recording medium storing computer program controlling the apparatus |
Country Status (5)
Country | Link |
---|---|
US (1) | US8831872B2 (en) |
EP (2) | EP1898291B1 (en) |
JP (1) | JP4886302B2 (en) |
KR (1) | KR100776215B1 (en) |
DE (2) | DE602006007247D1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090125175A1 (en) * | 2007-11-09 | 2009-05-14 | Samsung Electronics Co., Ltd. | Apparatus and method for generating three-dimensional map using structured light |
US9886036B2 (en) * | 2014-02-10 | 2018-02-06 | John Bean Technologies Corporation | Routing of automated guided vehicles |
US10373335B1 (en) * | 2014-07-10 | 2019-08-06 | Hrl Laboratories, Llc | System and method for location recognition and learning utilizing convolutional neural networks for robotic exploration |
US10970872B2 (en) * | 2016-02-25 | 2021-04-06 | Tectmion Research & Development Foundation Limited | System and method for image capture device pose estimation |
Families Citing this family (68)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11835343B1 (en) | 2004-08-06 | 2023-12-05 | AI Incorporated | Method for constructing a map while performing work |
KR100776215B1 (en) | 2005-01-25 | 2007-11-16 | 삼성전자주식회사 | Apparatus and method for estimating location and generating map of mobile body, using upper image, computer-readable recording media storing computer program controlling the apparatus |
US8930023B2 (en) * | 2009-11-06 | 2015-01-06 | Irobot Corporation | Localization by learning of wave-signal distributions |
US20070265773A1 (en) * | 2006-05-09 | 2007-11-15 | Hately Andrew D | User friendly method of computer generating maps for regional delivery services |
KR100757937B1 (en) * | 2006-06-30 | 2007-09-12 | 삼성전자주식회사 | Simultaneous localization and map building method for robot |
JPWO2008026463A1 (en) * | 2006-08-30 | 2010-01-21 | 日本電気株式会社 | Position measurement system, robot, position measurement method, and sound source position calculation program |
KR100773271B1 (en) * | 2006-10-25 | 2007-11-05 | 전자부품연구원 | Method for localization of mobile robot with a single camera |
KR100792852B1 (en) * | 2006-10-25 | 2008-01-14 | 전자부품연구원 | Method for extracting distance of landmark of mobile robot with a single camera |
JP2008137551A (en) * | 2006-12-04 | 2008-06-19 | Aisin Seiki Co Ltd | Vehicular seat device |
JP2008185417A (en) * | 2007-01-29 | 2008-08-14 | Sony Corp | Information processing device, information processing method, and computer program |
JP4802112B2 (en) * | 2007-02-08 | 2011-10-26 | 株式会社東芝 | Tracking method and tracking device |
KR100866380B1 (en) * | 2007-02-13 | 2008-11-03 | 한국과학기술연구원 | Method for esrimating location using ObjectionRecognition of a robot |
KR100877071B1 (en) | 2007-07-18 | 2009-01-07 | 삼성전자주식회사 | Method and apparatus of pose estimation in a mobile robot based on particle filter |
KR101337534B1 (en) * | 2007-07-24 | 2013-12-06 | 삼성전자주식회사 | Apparatus and method for localization of moving robot |
KR101503904B1 (en) | 2008-07-07 | 2015-03-19 | 삼성전자 주식회사 | Apparatus and method for building map for mobile robot |
KR101538775B1 (en) * | 2008-09-12 | 2015-07-30 | 삼성전자 주식회사 | Apparatus and method for localization using forward images |
KR101591471B1 (en) * | 2008-11-03 | 2016-02-04 | 삼성전자주식회사 | apparatus and method for extracting feature information of object and apparatus and method for generating feature map |
KR101141601B1 (en) * | 2008-12-01 | 2012-05-17 | 한국전자통신연구원 | Quick map building apparatus based on vision natural landmark |
KR101524020B1 (en) * | 2009-03-06 | 2015-05-29 | 엘지전자 주식회사 | Method for gradually building map by mobile robot and correcting position of mobile robot |
JP5227251B2 (en) * | 2009-05-01 | 2013-07-03 | 日本放送協会 | Camera calibration device and camera calibration program |
KR101540666B1 (en) * | 2009-07-22 | 2015-07-31 | 엘지전자 주식회사 | Apparatus and Method for detecting Rotation-Invariant Features for Localization of a mobile robot |
US8744665B2 (en) * | 2009-07-28 | 2014-06-03 | Yujin Robot Co., Ltd. | Control method for localization and navigation of mobile robot and mobile robot using the same |
WO2011044298A2 (en) * | 2009-10-06 | 2011-04-14 | Escrig M Teresa | Systems and methods for establishing an environmental representation |
KR101314588B1 (en) * | 2009-10-26 | 2013-10-07 | 한국전자통신연구원 | Method and apparatus for producing map of artificial mark, method and apparatus for measuring position of mobile object by using same |
CN102656532B (en) * | 2009-10-30 | 2015-11-25 | 悠进机器人股份公司 | For ground map generalization and the update method of position of mobile robot identification |
US20110106783A1 (en) * | 2009-11-03 | 2011-05-05 | Kalbhavi Srinivas | Mobile search with a landmark |
US8180146B2 (en) * | 2009-12-22 | 2012-05-15 | The Chinese University Of Hong Kong | Method and apparatus for recognizing and localizing landmarks from an image onto a map |
KR101167627B1 (en) * | 2010-02-25 | 2012-07-23 | 연세대학교 산학협력단 | Apparatus and Method for Double-Updating in simultaneous localization and mapping for a mobile robot |
US8406506B2 (en) * | 2010-05-18 | 2013-03-26 | Honda Motor Co., Ltd. | Fast sub-pixel optical flow estimation |
KR101677634B1 (en) * | 2010-07-12 | 2016-11-18 | 엘지전자 주식회사 | Robot cleaner and controlling method of the same |
AU2011305154B2 (en) * | 2010-09-24 | 2015-02-05 | Irobot Corporation | Systems and methods for VSLAM optimization |
KR101305405B1 (en) * | 2011-03-23 | 2013-09-06 | (주)하기소닉 | Method for Localizing Intelligent Mobile Robot by using a lateral landmark |
KR101311100B1 (en) * | 2011-08-27 | 2013-09-25 | 고려대학교 산학협력단 | Method for recognizing the self position of a mobile robot unit using arbitrary ceiling features on the ceiling image/feature map |
KR101280392B1 (en) * | 2011-08-29 | 2013-07-01 | 연세대학교 산학협력단 | Apparatus for managing map of mobile robot based on slam and method thereof |
EP2776216B1 (en) * | 2011-11-11 | 2022-08-31 | iRobot Corporation | Robot apparautus and control method for resuming operation following a pause. |
CN103208111B (en) * | 2012-01-17 | 2015-10-07 | 富士通株式会社 | For method and apparatus and the image processing equipment of correction image angle point |
CN103292789B (en) * | 2012-02-27 | 2015-12-09 | 联想(北京)有限公司 | A kind of localization method and electronic equipment |
CN103680291B (en) * | 2012-09-09 | 2016-12-21 | 复旦大学 | The method synchronizing location and mapping based on ceiling vision |
JP6233706B2 (en) | 2013-04-02 | 2017-11-22 | パナソニックIpマネジメント株式会社 | Autonomous mobile device and self-position estimation method of autonomous mobile device |
WO2015049717A1 (en) * | 2013-10-01 | 2015-04-09 | 株式会社日立製作所 | Device for estimating position of moving body and method for estimating position of moving body |
CN103901891A (en) * | 2014-04-12 | 2014-07-02 | 复旦大学 | Dynamic particle tree SLAM algorithm based on hierarchical structure |
EP3161572B1 (en) * | 2014-06-27 | 2019-01-23 | Crown Equipment Corporation | Lost vehicle recovery utilizing associated feature pairs |
WO2016005011A1 (en) * | 2014-07-10 | 2016-01-14 | Aktiebolaget Electrolux | Method in a robotic cleaning device for facilitating detection of objects from captured images |
US11051000B2 (en) * | 2014-07-14 | 2021-06-29 | Mitsubishi Electric Research Laboratories, Inc. | Method for calibrating cameras with non-overlapping views |
KR101600862B1 (en) | 2014-08-26 | 2016-03-08 | 연세대학교 산학협력단 | stereo vision system using a plurality of uav |
JP6397355B2 (en) * | 2015-02-26 | 2018-09-26 | 本田技研工業株式会社 | Control device for moving body |
JP6651295B2 (en) * | 2015-03-23 | 2020-02-19 | 株式会社メガチップス | Moving object control device, program and integrated circuit |
CN104964683B (en) * | 2015-06-04 | 2018-06-01 | 上海物景智能科技有限公司 | A kind of closed-loop corrected method of indoor environment map building |
CN105333869A (en) * | 2015-11-04 | 2016-02-17 | 天津津航计算技术研究所 | Unmanned reconnaissance aerial vehicle synchronous positioning and picture compositing method based on self-adaption EKF |
CN105606092B (en) * | 2016-02-04 | 2019-02-15 | 中国科学院电子学研究所 | A kind of Position Method for Indoor Robot and system |
CN105737832B (en) * | 2016-03-22 | 2019-03-22 | 北京工业大学 | Distributed SLAM method based on global optimum's data fusion |
KR102192791B1 (en) * | 2016-04-25 | 2020-12-18 | 엘지전자 주식회사 | Mobile robot and control method of mobile robot |
WO2018020853A1 (en) * | 2016-07-29 | 2018-02-01 | Necソリューションイノベータ株式会社 | Mobile body control system, control signal transmission system, mobile body control method, program, and recording medium |
CN106502253B (en) * | 2016-12-22 | 2019-11-22 | 深圳乐动机器人有限公司 | A kind of mobile device independently builds drawing method and device entirely |
US10824888B1 (en) * | 2017-01-19 | 2020-11-03 | State Farm Mutual Automobile Insurance Company | Imaging analysis technology to assess movements of vehicle occupants |
CN109147374A (en) * | 2017-06-15 | 2019-01-04 | 深圳市金溢科技股份有限公司 | Parking lot management method, system, on board unit and background server |
CN107563308B (en) * | 2017-08-11 | 2020-01-31 | 西安电子科技大学 | SLAM closed loop detection method based on particle swarm optimization algorithm |
US11274929B1 (en) * | 2017-10-17 | 2022-03-15 | AI Incorporated | Method for constructing a map while performing work |
US10422648B2 (en) | 2017-10-17 | 2019-09-24 | AI Incorporated | Methods for finding the perimeter of a place using observed coordinates |
US10762658B2 (en) * | 2017-10-24 | 2020-09-01 | Altek Corporation | Method and image pick-up apparatus for calculating coordinates of object being captured using fisheye images |
KR102221695B1 (en) * | 2018-01-15 | 2021-03-02 | 에스케이텔레콤 주식회사 | Apparatus and method for updating high definition map for autonomous driving |
EP3806036A4 (en) * | 2018-06-07 | 2022-03-16 | Uisee Technologies (Beijing) Ltd | Simultaneous localization and mapping method and apparatus |
KR102147211B1 (en) * | 2018-10-22 | 2020-08-24 | 엘지전자 주식회사 | Controlling method for Artificial intelligence Moving robot |
CN109343540A (en) * | 2018-11-30 | 2019-02-15 | 广东工业大学 | A kind of rear end SLAM track optimizing method based on winding detection |
KR102171228B1 (en) * | 2019-01-31 | 2020-10-28 | 인하대학교 산학협력단 | Method and Apparatus for Finger Printing-based Positioning in Urban Canyon Environment by using a Wide Angle Camera |
PH12019050076A1 (en) * | 2019-05-06 | 2020-12-02 | Samsung Electronics Co Ltd | Enhancing device geolocation using 3d map data |
US10854012B1 (en) * | 2019-05-29 | 2020-12-01 | Dell Products, L.P. | Concealing loss of distributed simultaneous localization and mapping (SLAM) data in edge cloud architectures |
JP2022050929A (en) * | 2020-09-18 | 2022-03-31 | カシオ計算機株式会社 | Mobile object positioning device, mobile object positioning method and program |
Citations (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS6089213A (en) | 1983-10-19 | 1985-05-20 | Komatsu Ltd | Detecting method for position and direction of unmanned truck |
US5051906A (en) | 1989-06-07 | 1991-09-24 | Transitions Research Corporation | Mobile robot navigation employing retroreflective ceiling features |
JPH10177414A (en) | 1996-12-16 | 1998-06-30 | Matsushita Electric Ind Co Ltd | Device for recognizing traveling state by ceiling picture |
US5793934A (en) | 1994-06-22 | 1998-08-11 | Siemens Aktiengesellschaft | Method for the orientation, route planning and control of an autonomous mobile unit |
JPH11259659A (en) | 1998-03-11 | 1999-09-24 | Hokkaido Prefecture | Environment map producing method by mobile robot |
US6285393B1 (en) | 1993-09-08 | 2001-09-04 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
US20020067432A1 (en) * | 2000-10-20 | 2002-06-06 | Satoshi Kondo | Block distortion detection method, block distortion detection apparatus, block distortion removal method, and block distortion removal apparatus |
US20020153184A1 (en) | 2001-04-18 | 2002-10-24 | Jeong-Gon Song | Robot cleaner, robot cleaning system and method for controlling same |
GB2382251A (en) | 2000-11-17 | 2003-05-21 | Samsung Kwangju Electronics Co | Mobile Robot |
US20030216834A1 (en) | 2000-05-01 | 2003-11-20 | Allard James R. | Method and system for remote control of mobile robot |
JP2004012429A (en) | 2002-06-11 | 2004-01-15 | Mitsubishi Heavy Ind Ltd | Self-position/attitude identification device and self-position/attitude identification method |
WO2004059900A2 (en) | 2002-12-17 | 2004-07-15 | Evolution Robotics, Inc. | Systems and methods for visual simultaneous localization and mapping |
JP2004276168A (en) | 2003-03-14 | 2004-10-07 | Japan Science & Technology Agency | Map making system for mobile robot |
JP2004355601A (en) | 2003-05-07 | 2004-12-16 | Mitsubishi Space Software Kk | Target chasing device, target chasing method, computer-readable recording medium with program recorded and program |
KR20060013022A (en) | 2004-08-05 | 2006-02-09 | 삼성전자주식회사 | Simultaneous localization and map building method for robot |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0953939A (en) * | 1995-08-18 | 1997-02-25 | Fujitsu Ltd | Method and system for measuring position of mobile vehicle |
JP3237705B2 (en) | 1999-02-04 | 2001-12-10 | 日本電気株式会社 | Obstacle detection device and moving object equipped with obstacle detection device |
US6374155B1 (en) | 1999-11-24 | 2002-04-16 | Personal Robotics, Inc. | Autonomous multi-platform robot system |
KR100483548B1 (en) * | 2002-07-26 | 2005-04-15 | 삼성광주전자 주식회사 | Robot cleaner and system and method of controlling thereof |
JP2004030445A (en) * | 2002-06-27 | 2004-01-29 | National Institute Of Advanced Industrial & Technology | Method, system, and program for estimating self-position of moving robot |
JP4042517B2 (en) * | 2002-10-09 | 2008-02-06 | 株式会社日立製作所 | Moving body and position detection device thereof |
KR100641491B1 (en) * | 2002-12-30 | 2006-10-31 | 동부일렉트로닉스 주식회사 | Method for making deep detail contact plug in semiconductor |
KR100557682B1 (en) * | 2003-07-21 | 2006-03-07 | 주식회사 한울로보틱스 | Locomotion methode of mobile robot using floor image informations |
KR100776215B1 (en) | 2005-01-25 | 2007-11-16 | 삼성전자주식회사 | Apparatus and method for estimating location and generating map of mobile body, using upper image, computer-readable recording media storing computer program controlling the apparatus |
-
2005
- 2005-01-25 KR KR1020050006573A patent/KR100776215B1/en active IP Right Grant
-
2006
- 2006-01-13 US US11/331,058 patent/US8831872B2/en not_active Expired - Fee Related
- 2006-01-23 DE DE602006007247T patent/DE602006007247D1/en active Active
- 2006-01-23 EP EP07150373A patent/EP1898291B1/en not_active Expired - Fee Related
- 2006-01-23 EP EP06250337A patent/EP1684143B1/en not_active Expired - Fee Related
- 2006-01-23 DE DE602006019726T patent/DE602006019726D1/en active Active
- 2006-01-25 JP JP2006016141A patent/JP4886302B2/en not_active Expired - Fee Related
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS6089213A (en) | 1983-10-19 | 1985-05-20 | Komatsu Ltd | Detecting method for position and direction of unmanned truck |
US5051906A (en) | 1989-06-07 | 1991-09-24 | Transitions Research Corporation | Mobile robot navigation employing retroreflective ceiling features |
US6285393B1 (en) | 1993-09-08 | 2001-09-04 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
US5793934A (en) | 1994-06-22 | 1998-08-11 | Siemens Aktiengesellschaft | Method for the orientation, route planning and control of an autonomous mobile unit |
JPH10177414A (en) | 1996-12-16 | 1998-06-30 | Matsushita Electric Ind Co Ltd | Device for recognizing traveling state by ceiling picture |
JPH11259659A (en) | 1998-03-11 | 1999-09-24 | Hokkaido Prefecture | Environment map producing method by mobile robot |
US20030216834A1 (en) | 2000-05-01 | 2003-11-20 | Allard James R. | Method and system for remote control of mobile robot |
US20020067432A1 (en) * | 2000-10-20 | 2002-06-06 | Satoshi Kondo | Block distortion detection method, block distortion detection apparatus, block distortion removal method, and block distortion removal apparatus |
GB2382251A (en) | 2000-11-17 | 2003-05-21 | Samsung Kwangju Electronics Co | Mobile Robot |
US20020153184A1 (en) | 2001-04-18 | 2002-10-24 | Jeong-Gon Song | Robot cleaner, robot cleaning system and method for controlling same |
JP2004012429A (en) | 2002-06-11 | 2004-01-15 | Mitsubishi Heavy Ind Ltd | Self-position/attitude identification device and self-position/attitude identification method |
WO2004059900A2 (en) | 2002-12-17 | 2004-07-15 | Evolution Robotics, Inc. | Systems and methods for visual simultaneous localization and mapping |
US20040167669A1 (en) * | 2002-12-17 | 2004-08-26 | Karlsson L. Niklas | Systems and methods for using multiple hypotheses in a visual simultaneous localization and mapping system |
JP2004276168A (en) | 2003-03-14 | 2004-10-07 | Japan Science & Technology Agency | Map making system for mobile robot |
JP2004355601A (en) | 2003-05-07 | 2004-12-16 | Mitsubishi Space Software Kk | Target chasing device, target chasing method, computer-readable recording medium with program recorded and program |
KR20060013022A (en) | 2004-08-05 | 2006-02-09 | 삼성전자주식회사 | Simultaneous localization and map building method for robot |
Non-Patent Citations (20)
Title |
---|
Chris Harris et al., "A Combined Corner and Edge Detector", The Plessey Company, 1988, pp. 147-152. |
Davison, A. et al., "Real-Time 3D Slam with Wide-Angle Vision," 5th IFAC/EURON Symposium on Intelligent Autonomous Vehicles, Jul. 2004. |
Extended European Search Report dated May 24, 2006 issued by the EPO in European Patent Application 06250337.0. |
Fiala, M. et al., "Robot Navigation Using Panoramic Tracking," Pattern Recognition, vol. 37, No. 11, pp. 2195-2215, Nov. 2004. |
Guivant, J. et al., "Simultaneous Localization and Map Building Using Natural Features and Absolute Information," Robotics and Autonomous Systems, vol. 984, pp. 1-12, 2002. |
Hajdiab Het al:"Vision Based Robot Localization"Haptic, Audio and Visual Environments and Their Applications, 2003. Have 2003. Proceedings. The 2ndIEEE International Workshop on Sep. 20-21, 2003, Piscatwaway, NJ, USA, IEEE Sep. 20, 200. * |
Hajjdiab, H. et al., "Vision-Based Robot Localization," 2nd IEEE International Workshop on Audio and Visual Environments and Their Applications, pp. 19-24, Sep. 2003. |
Harris, C. et al., "A Combined Corner and Edge Detector," Plessey Research Roke Manor, pp. 147-151, 1998. |
Harris, Chris "A Combined Corner and Edge Detector", Plessy Research Roke Manor, Unied Kingdom, The Plessey Company plc, 1988. * |
Japanese Office Action dated Jul. 7, 2009 issued in Japanese Patent Application No. 2006-016141. |
Japanese Office Action issued Jan. 11, 2011 in corresponding Japanese Patent Application 2006-016141. |
Krystian Mikolajczyk et al., "A performance evaluation of local descriptors", Oct. 2004, 33 pages. |
Lang, S. et al., "Visual Correction of Orientation Error for a Mobile Robot," IEEE International Conference on Intelligent Robots and Systems, pp. 1328-1333, 1999. |
Lowe, D., "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, pp. 1-28, 2004. |
Lucas, B. et al., "An Iterative Image Registration Technique with an Application to Stereo Vision," Proceedings of Imaging Understanding Workshop, pp. 121-130, 1981. |
M.W.M.G. Dissanayake et al., "A Solution to the Simultaneous Localisation and Map Building (SLAM) Problem", 2006, 14 pages. |
Robot Navigation Using Panoramic Tracking by Fiala et al., vol. 37, No. 11, Nov. 2004, pp. 2195-2215. |
Se, S. et al., "Mobile Robot Localization and Mapping with Uncertainty using Scale-Invariant Visual Landmarks," International Journal of Robotics Research, vol. 21, No. 8, pp. 735-758, Aug. 2002. |
Thrun, S. et al., "Robust Monte Carlo Localization for Mobile Robots," Artificial Intelligence, pp. 1-49, Feb. 2001. |
Vision-Based Robot Localization by Hajdiab et al., 2nd IEEE International Workshop on Sep. 20, 2003, pp. 19-24. |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090125175A1 (en) * | 2007-11-09 | 2009-05-14 | Samsung Electronics Co., Ltd. | Apparatus and method for generating three-dimensional map using structured light |
US9182763B2 (en) * | 2007-11-09 | 2015-11-10 | Samsung Electronics Co., Ltd. | Apparatus and method for generating three-dimensional map using structured light |
US9886036B2 (en) * | 2014-02-10 | 2018-02-06 | John Bean Technologies Corporation | Routing of automated guided vehicles |
US10373335B1 (en) * | 2014-07-10 | 2019-08-06 | Hrl Laboratories, Llc | System and method for location recognition and learning utilizing convolutional neural networks for robotic exploration |
US10970872B2 (en) * | 2016-02-25 | 2021-04-06 | Tectmion Research & Development Foundation Limited | System and method for image capture device pose estimation |
Also Published As
Publication number | Publication date |
---|---|
EP1898291A1 (en) | 2008-03-12 |
DE602006019726D1 (en) | 2011-03-03 |
EP1898291B1 (en) | 2011-01-19 |
JP2006209770A (en) | 2006-08-10 |
US20060165276A1 (en) | 2006-07-27 |
KR20060085751A (en) | 2006-07-28 |
KR100776215B1 (en) | 2007-11-16 |
DE602006007247D1 (en) | 2009-07-30 |
JP4886302B2 (en) | 2012-02-29 |
EP1684143A1 (en) | 2006-07-26 |
EP1684143B1 (en) | 2009-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8831872B2 (en) | Apparatus and method for estimating location of mobile body and generating map of mobile body environment using upper image of mobile body environment, and computer readable recording medium storing computer program controlling the apparatus | |
US11561549B2 (en) | Method of controlling mobile robot | |
Jeong et al. | CV-SLAM: A new ceiling vision-based SLAM technique | |
Nobili et al. | Predicting alignment risk to prevent localization failure | |
KR20220065672A (en) | Deep smartphone sensors fusion for indoor positioning and tracking | |
Carrera et al. | Lightweight SLAM and Navigation with a Multi-Camera Rig. | |
WO2020194079A1 (en) | Method and system for performing localization of an object in a 3d | |
KR101002776B1 (en) | Apparatus and method for estimating location and generating map of mobile body, using upper image, computer-readable recording media storing computer program controlling the apparatus | |
Nandkumar et al. | Simulation of Indoor Localization and Navigation of Turtlebot 3 using Real Time Object Detection | |
Kim et al. | Vision-based navigation with efficient scene recognition | |
Zou et al. | Static map reconstruction and dynamic object tracking for a camera and laser scanner system | |
Muravyev et al. | Evaluation of Topological Mapping Methods in Indoor Environments | |
Klippenstein et al. | Performance evaluation of visual SLAM using several feature extractors | |
Basit et al. | Joint localization and target tracking with a monocular camera | |
Roussel et al. | Deep-geometric 6 dof localization from a single image in topo-metric maps | |
Wang et al. | Improved simultaneous localization and mapping by stereo camera and SURF | |
Baligh Jahromi et al. | a Preliminary Work on Layout Slam for Reconstruction of Indoor Corridor Environments | |
Gil et al. | Building Visual Maps with a Single Omnidirectional Camera. | |
Słowak et al. | Homography augmented particle filter SLAM | |
Ivanchenko et al. | Elevation-based MRF stereo implemented in real-time on a GPU | |
dos Santos Fernandes et al. | Visual and inertial data fusion for Globally consistent point cloud registration | |
Sundaresan | Pixel-based object motion detection and tracking with a moving camera | |
Gamallo et al. | Visual FastSLAM through omnivision | |
Wang | Lidar-based Localization and Pose Estimation Approaches in the Map Built by Monocular SLAM | |
Kirnos et al. | Landmarks detection by contour analysis in the problem of SLAM |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HONG, SUNGI;LEE, HYOUNGKI;MYEONG, HYEON;REEL/FRAME:017477/0934 Effective date: 20060105 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551) Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20220909 |