JP2017207942A - Image processing apparatus, self position estimation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve an estimation accuracy of a self position.SOLUTION: An image processing apparatus includes: a feature point tracking part configured to track a feature point extracted from an image of a photographed subject; a mark recognition part configured to recognize a mark previously arranged in the image; and a self position estimation part configured to estimate a first self position according to a tracking result of the feature point, estimate a second self position according to the mark, and adopt one of the first self position and the second self position as a self position according to predetermined conditions.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing device, a self-position estimation method, and a program.

  As a technique for allowing a mobile body such as an automobile and a robot to autonomously operate, a technique for autonomously building a map and estimating a self-position in an unknown environment (for example, SLAM: Simulaneous Localization And Mapping) is known.

  In the above-described technique, a plurality of feature points are extracted from a plurality of time-series images taken with a camera at different times, and the self-position is estimated by a vector obtained by tracking the plurality of feature points (for example, optical flow). To do. However, tracking feature points in time series accumulates tracking errors and increases self-position estimation errors. In such a case, a technique for estimating the self-position using an RFID and a landmark is known.

  However, in the above-described technique, when the camera is moving relative to the landmark, if the self-position is estimated from an image obtained by capturing the landmark, an error occurs due to the effect of motion blur and the self-position estimation accuracy is reduced. There is a problem of low.

  The present invention has been made in view of the above, and an object thereof is to improve self-position estimation accuracy.

  In order to solve the above-described problems and achieve the object, the present invention provides an image processing apparatus, a feature point tracking unit that tracks a feature point of the subject extracted from an image of the photographed subject, A mark recognizing unit for recognizing a predetermined mark, a first self-position is estimated based on the tracking result of the feature point, a second self-position is estimated based on the mark, and a predetermined condition is satisfied. And a self-position estimating unit that employs one of the first self-position and the second self-position as a self-position.

  The present invention can improve the self-position estimation accuracy.

FIG. 1 is an overall configuration diagram of an image processing system having an image processing apparatus according to an embodiment. FIG. 2 is a block diagram illustrating functions of the image processing apparatus. FIG. 3 is a diagram illustrating an example of an image that is a frame image. FIG. 4 is a part of a diagram illustrating an example of an image that is the (n−1) -th frame image. FIG. 5 is a part of a diagram illustrating an example of an image that is an nth frame image. FIG. 6 is a diagram illustrating the tracking of feature points between the (n−1) th image and the nth image. FIG. 7 is a diagram for explaining the coordinate system handled in mark recognition and camera position estimation. FIG. 8 is a diagram for explaining the coordinate system and camera position estimation by the pinhole camera. FIG. 9 is a diagram for explaining an error in the left-right direction (or horizontal direction) in the estimation of the camera position by the mark. FIG. 10 is a diagram for explaining an error in the vertical direction (or vertical direction) in the estimation of the camera position by the mark. FIG. 11 is a flowchart of self-position estimation processing by the image processing apparatus. FIG. 12 is a diagram illustrating another self-position estimation. FIG. 13 is a plan view seen from above for explaining the estimation of the self-position of a movable object that can fly. FIG. 14 is a side view for explaining the estimation of the self-position of a movable object that can fly. FIG. 15 is a block diagram schematically illustrating a hardware configuration of the image processing apparatus according to the present embodiment.

  Similar components are included in the following exemplary embodiments and modifications. Therefore, below, the same code | symbol is attached | subjected to the same component, and the overlapping description is partially abbreviate | omitted. Portions included in the embodiments and modifications can be configured by replacing corresponding portions in other embodiments and modifications. In addition, the configuration, position, and the like of the parts included in the embodiments and modifications are the same as those in the other embodiments and modifications unless otherwise specified.

<Embodiment>
FIG. 1 is an overall configuration diagram of an image processing system 10 having an image processing apparatus 16 according to an embodiment. As shown in FIG. 1, the image processing system 10 includes a mark MK, a camera 14, and an image processing device 16.

  The mark MK is also called a landmark, and is a figure or the like provided in advance on a building 90 such as a wall or a pillar of a parking lot or the like. The mark MK is provided at a position where it can be photographed by the camera 14. The mark MK may be a sign, a character, a number, or the like that can be easily recognized by the image processing device 16.

  The camera 14 is provided on a moving body 92 such as an automobile. The moving body 92 may be an airplane, a motorcycle, an electric product such as a vacuum cleaner, an agricultural tool, or the like. The camera 14 is provided in front of the moving body 92 indicated by an arrow, for example, a rearview mirror. The camera 14 captures a subject in front of the moving body 92 and generates an image (for example, a frame image). The camera 14 may be provided behind the moving body 92. In this case, the camera 14 captures a subject behind the moving body 92 and generates an image.

  The image processing device 16 may be provided in the moving body 92 together with the camera 14, or may be provided in a place different from the moving body 92. The image processing device 16 is connected to the camera 14 so that information can be transmitted and received. The image processing device 16 receives an image from the camera 14. The image processing device 16 extracts the feature point PT of the building 90 or the like from the received image, recognizes the mark MK, and estimates the self position SP that is the self position. The feature point PT is a point that is conspicuously recognized, such as a corner of a subject such as a building 90 included in the image.

  FIG. 2 is a block diagram illustrating functions of the image processing device 16. As shown in FIG. 2, the image processing device 16 includes an image acquisition unit 20, a frame memory 22, an image correction unit 24, an image memory 26, a feature point extraction unit 28, a feature point storage memory 30, and a feature. A point tracking unit 32, a tracking result storage memory 34, a mark recognition unit 36, a mark information storage memory 38, a mark recognition result storage memory 40, and a self-position estimation unit 42 are provided. Image acquisition unit 20, frame memory 22, image correction unit 24, image memory 26, feature point extraction unit 28, feature point storage memory 30, feature point tracking unit 32, tracking result storage memory 34, mark recognition unit 36, mark information storage The memory 38, the mark recognition result storage memory 40, and the self-position estimation unit 42 may be configured by software, may be configured by hardware, a part is configured by software, and the rest is configured by hardware. Also good.

  The image acquisition unit 20 is connected to the camera 14. The image acquisition unit 20 acquires an image IG such as a frame image captured by the camera 14. The image acquisition unit 20 stores the image IG in the frame memory 22. The image acquisition unit 20 outputs information about the image IG stored in the frame memory 22 (for example, the number of lines of the stored image IG) and the like to the image correction unit 24.

  The image correction unit 24 reads the image IG from the frame memory 22 based on information such as the number of lines related to the image IG acquired from the image acquisition unit 20. Note that the image correction unit 24 may read a part of the image IG instead of the entire image IG from the frame memory 22. For example, the image correction unit 24 may read the image IG when the image IG having a predetermined number of lines is stored in the frame memory 22. The image correction unit 24 executes correction processing such as distortion correction on the read image IG. The image correction unit 24 stores the corrected image IG in the image memory 26.

  The feature point extraction unit 28 reads the corrected image IG from the image memory 26. Note that the feature point extraction unit 28 may read a part of the image IG from the image memory 26 instead of the entire image IG. For example, when a part of the image IG having a predetermined number of lines in the image IG is stored in the image memory 26, the feature point extraction unit 28 may read the stored part of the image IG. . The feature point extraction unit 28 extracts one or a plurality of feature points PT from the read image IG of the subject such as the building 90. The feature point extraction unit 28 extracts the feature point PT from the image IG by a corner detection method such as known Harris or Moravec (for example, http://en.wikipedia.org/wiki/ % E3% 82% B3% E3% 83% BC% E3% 83% 8A% E3% 83% BC% E6% A4% 9C% E5% 87% BA% E6% B3% 95). The feature point extraction unit 28 stores the extracted feature points PT in the feature point storage memory 30.

  The feature point tracking unit 32 tracks the feature point PT of the subject extracted by the feature point extraction unit 28 from the photographed image IG of the subject such as the building 90. For example, the feature point tracking unit 32 reads out the feature point PT extracted from the previous image IG from the tracking result storage memory 34. The feature point tracking unit 32 acquires, from the image memory 26, an image IG in a predetermined necessary pixel range included in the previous image IG. The feature point tracking unit 32 creates a template patch TP, which will be described later, based on the feature point PT and the previous image IG. The feature point tracking unit 32 reads the current image IG from the image memory 26. The feature point tracking unit 32 searches and tracks the feature point PT by block matching based on the template patch TP and the current image IG. When the feature point tracking unit 32 can track the feature point PT, the tracking result storage memory 34 updates the tracking result including the optical flow FL, which is a result indicating that the feature point PT has been tracked. On the other hand, when the feature point tracking unit 32 cannot track the feature point PT, the feature point tracking unit 32 stores the feature point PT read from the feature point storage memory 30 in the tracking result storage memory 34.

  The mark recognition unit 36 recognizes a mark MK provided in advance on a subject such as a building 90 in the image IG. For example, when the feature point PT is stored in the feature point storage memory 30, the mark recognition unit 36 reads pattern information and position information set in advance for the mark MK from the mark information storage memory 38, and at the same time, the feature point storage memory 30. To obtain the feature point PT. The mark recognition unit 36 recognizes the mark MK in the image IG based on the feature point PT and the pattern information. The mark recognition unit 36 stores the recognition result and position information of the mark MK in the mark recognition result storage memory 40. Note that the mark recognition unit 36 may execute the process of recognizing the mark MK simultaneously with the process of the feature point tracking unit 32 tracking the feature point PT.

  The self-position estimation unit 42 reads the tracking result from the tracking result storage memory 34. The self-position estimating unit 42 estimates the first self-position based on the tracking result. The self-position estimation unit 42 reads the recognition result and position information of the mark MK from the mark recognition result storage memory 40. The self-position estimating unit 42 estimates the second self-position based on the recognized mark MK. For example, the self-position estimating unit 42 estimates the second self-position based on preset position information of the mark MK. The self-position estimating unit 42 employs one of the two estimated first self-positions and second self-positions as the self-position SP based on a predetermined condition.

  FIG. 3 is a diagram illustrating an example of an image IG0 that is a frame image. FIG. 4 is a diagram illustrating an example of a part of the image IG1 which is the (n−1) th frame image. FIG. 5 is a diagram illustrating an example of a part of the image IG2 which is the nth frame image. FIG. 6 is a diagram for explaining the tracking of the feature point PT between the (n−1) th image IG1 and the nth image IG2. 3 to 6 are examples of the image IG in a state where the moving body 92 is traveling straight on the road 94.

  The case where the image acquisition unit 20 acquires the nth image IG0 shown in FIG. 3 from the camera 14 and stores the image IG0 corrected by the image correction unit 24 in the image memory 26 will be described. In this case, for example, the feature point extraction unit 28 extracts the feature points PT1 and PT2 from the (n−1) th image IG1 illustrated in FIG. 4 and stores them in the feature point storage memory 30. The feature points PT1 and PT2 are, for example, corners (or corners) of the buildings 90A and 90B near the road 94.

  The feature point tracking unit 32 acquires the (n−1) th feature points PT1 and PT2 from the tracking result storage memory 34. The feature point tracking unit 32 extracts, from the image memory 26, the image IG1 shown in FIG. 4 in the range of several lines around the (n-1) th feature point PT1 and PT2 in the (n-1) th image IG0. read out. The feature point tracking unit 32 converts the rectangular template patches TP1 and TP2 in the range of several pixels left and right and up and down around the (n-1) th feature point PT1 and PT2 to the (n-1) th image IG1. Extract from Template patches TP1 and TP2 may be circular.

  The feature point tracking unit 32 acquires the nth image IG2 shown in FIG. The feature point tracking unit 32 searches the nth image IG2 for a region similar to the template patches TP1 and TP2 by block matching using the similarity. For example, the feature point tracking unit 32 performs pattern matching (for example, normalized correlation) or template matching (for example, see http://imagingsolution.blog107.fc2.com/blog-entry-186.html) as block matching. Execute. The feature point tracking unit 32 may execute a search by block matching in the area AR1 including the position PT1 'having the same coordinates as the feature point PT1 in the image IG2. In the area AR1, the feature point tracking unit 32 searches from the upper left end to the right, and after reaching the right end of the area AR1, moves to the left end one line below and searches again in the right direction. The feature point tracking unit 32 sequentially continues searching in the area AR1 of the image IG2 in this order (so-called raster scan). The feature point tracking unit 32 extracts regions similar to the (n−1) th template patches TP1 and TP2 from the nth image IG2 as template patches TP3 and TP4. The feature point tracking unit 32 sets the centers of the template patches TP3 and TP4 as the nth feature points PT3 and PT4. As shown in FIG. 6, the feature point tracking unit 32 uses the optical flows FL1 and FL2 which are vectors extending from the (n−1) th feature points PT1 and PT2 to the nth feature points PT3 and PT4 as tracking results. The nth feature points PT3 and PT4 are stored in the tracking result storage memory 34.

  FIG. 7 is a diagram for explaining the coordinate system handled in the recognition of the mark MK and the position estimation of the camera 14. The self-position estimation unit 42 includes a three-dimensional mark coordinate system (XmYmZm space shown in FIG. 7), a two-dimensional image coordinate system (uv space shown in FIG. 7), and a three-dimensional camera coordinate system (shown in FIG. 7). The mark MK is recognized based on the (XcYcZc space). The position of the mark MK is registered in advance in the mark information storage memory 38 for each mark MK. It is assumed that internal parameters of the camera 14 (focal length, principal point (image center), lens distortion coefficient, and the like of the camera 14) are known and registered in advance. The self-position estimation unit 42 estimates the position of the camera 14 on the mark coordinate system by inversely transforming the position of the mark MK and the internal parameters of the camera 14 (for example, http://www.hitl.washington.edu /artoolkit/documentation/tutorialcamera.htm and http://im-lab.net/artoolkit-overview/).

  FIG. 8 is a diagram for explaining the coordinate system based on the pinhole camera model and the position estimation of the camera 14. As shown in FIG. 8, the self-position estimation unit 42 may estimate the position and orientation of the camera 14 based on a pinhole camera model. The pinhole camera model is a three-dimensional world coordinate system (XwYwZw space shown in FIG. 8), a three-dimensional space model, for example, by a model representing a geometric relative relationship between a three-dimensional space and a two-dimensional image plane. A camera coordinate system (XcYcZc space shown in FIG. 8) and a two-dimensional image coordinate system (XiYi space shown in FIG. 8) are defined. The self-position estimation unit 42 solves the PnP problem based on the model based on the plurality of feature points PT and the internal parameters of the camera 14 calculated by the calibration, thereby obtaining external parameters (from the world coordinate system to the camera coordinate system). And the position of the camera 14 is estimated (see, for example, “OpenCV3 Programming Book / Yuichiro Fujimoto et al./Page 111 of Mynavi Publishing Co., Ltd.”).

  FIG. 9 is a diagram for explaining the error in the left-right direction (or horizontal direction) in the position estimation of the camera 14 by the mark MK. Errors in the position estimation of the camera 14 are caused by a rolling shutter phenomenon, motion blur, and the like. FIG. 9 is a plan view as seen from above. Xc and Zc shown on the camera 14 indicate coordinate axes of the camera coordinate system in FIG. A coordinate axis Xc indicates the horizontal direction of the moving body 92. A coordinate axis Zc indicates the front-rear direction of the moving body 92. In FIG. 9, the actual mark MK is present at a position PS0 in front of the camera 14 and the moving body 92, as indicated by the solid line. Xm and Zm shown on the mark MK indicate the mark coordinate system shown in FIG.

  The camera 14 attached to the moving body 92 captures the mark MK present at the front position PS0. The camera 14 may shoot the mark MK as a mark distorted in the left-right direction (for example, a rhombus) like the image MKa shown on the projection plane PPa due to the influence of a rolling shutter phenomenon or motion blur. The projection plane PPa is a plane on which the photographed mark MK is projected.

  In this case, the self-position estimation unit 42 erroneously recognizes that the mark MK exists at the position PS1 and erroneously estimates that the camera 14 exists at the position PS2 based on the image IG including the distorted image MKa. The self-position estimation unit 42 applies the vector VC1 indicated by the alternate long and short dash line calculated from the positional relationship between the position PS1 of the misrecognized mark MK and the position PS2 of the camera 14 based on the position PS0 of the known mark MK. , It is estimated that the camera 14 exists at the position PS3. Accordingly, the self-position estimation unit 42 estimates that the camera 14 exists at the position PS3 that is shifted in the left-right direction from the actual position PS4.

  In order for the self-position estimation unit 42 to delete these erroneously estimated results of the position PS3 of the camera 14, it is preferable to set in advance a determination range JR1 that is determined as a correct estimation result. The determination range JR1 is an example of a predetermined condition. The self-position estimation unit 42 employs the position of the camera 14 estimated to be within the determination range JR1 as the second self-position. The self-position estimation unit 42 discards the position of the camera 14 estimated to be outside the determination range JR1. The determination range JR1 is set, for example, by representing the position of the camera 14 viewed from the camera coordinate system with an Xm vector and a Zm vector.

  FIG. 10 is a diagram for explaining an error in the vertical direction (or vertical direction) in the position estimation of the camera 14 by the mark MK. Errors in the position estimation of the camera 14 are caused by a rolling shutter phenomenon, motion blur, and the like. FIG. 10 is a plan view seen from the substantially right direction. Yc and Zc shown on the camera 14 indicate coordinate axes of the camera coordinate system in FIG. A coordinate axis Yc indicates the vertical direction of the moving body 92. A coordinate axis Zc indicates the front-rear direction of the moving body 92. In FIG. 10, the actual mark MK is present at a position PS5 in front of the camera 14 and the moving body 92, as indicated by a solid line. Ym and Zm indicated on the mark MK indicate coordinate axes of the mark coordinate system shown in FIG.

  The camera 14 attached to the moving body 92 captures the mark MK present at the front position PS5. The camera 14 may shoot the mark MK in a vertically distorted shape (for example, a trapezoid) like the image MKb shown on the projection plane PPb due to the influence of a rolling shutter phenomenon or motion blur. The projection plane PPb is a plane on which the photographed mark MK is projected.

  In this case, the self-position estimation unit 42 erroneously recognizes that the mark MK exists at the position PS6 and erroneously estimates that the camera 14 exists at the position PS7. The self-position estimation unit 42 applies the vector VC2 calculated from the positional relationship between the position PS6 of the misrecognized mark MK and the position PS7 of the camera 14 with reference to the position PS5 of the known mark MK, so that the camera 14 Presumed to be at position PS8. Accordingly, the self-position estimating unit 42 estimates that the camera 14 exists at the position PS8 that is shifted in the vertical direction from the actual position PS9.

  In order for the self-position estimation unit 42 to delete these erroneously estimated position estimation results of the camera 14, it is preferable to set in advance a determination range JR <b> 2 that is determined to be a correct estimation result. The determination range JR2 is another example of a predetermined condition. The self-position estimation unit 42 employs the position of the camera 14 estimated to be within the determination range JR2 as the second self-position. The self-position estimating unit 42 discards the position of the camera 14 estimated to be outside the determination range JR2. The determination range JR2 is set, for example, by representing the position of the camera 14 viewed from the camera coordinate system with a Ym vector and a Zm vector.

  FIG. 11 is a flowchart of self-position estimation processing by the image processing device 16.

  As shown in FIG. 11, in the self-position estimation process, first, the image processing device 16 executes initialization (S102). For example, the self-position estimation unit 42 of the image processing device 16 sets the determination ranges JR1 and JR2. For example, the self-position estimation unit 42 sets a range in the Xc direction of the determination range JR1 shown in FIG. 9 with reference to the width of the road 94 on which the moving body 92 travels. The self-position estimating unit 42 sets a range in the Yc direction of the determination range JR2 illustrated in FIG. 10 with reference to the height of the camera 14 attached to the moving body 92 and the like.

  The image acquisition unit 20 acquires a frame image from the camera 14 as an image IG and stores it in the frame memory 22 (S104). The image correction unit 24 corrects the image IG stored in the frame memory 22 and stores it in the image memory 26 (S106).

  The feature point extraction unit 28 extracts the feature point PT from the corrected image IG stored in the image memory 26 and stores it in the feature point storage memory 30 (S108). The feature point tracking unit 32 generates a template patch TP from the previous feature point PT included in the tracking result, and tracks the feature point PT based on the corrected image IG stored in the image memory 26. (S110). The feature point tracking unit 32 stores the tracking result including the optical flow FL of the feature point PT in the tracking result storage memory 34 (S112).

  The mark recognition unit 36 recognizes the mark MK from the feature point group including the plurality of feature points PT stored in the feature point storage memory 30 based on the pattern information and position information of the mark MK acquired from the mark information storage memory 38. Then, the position information of the mark MK is stored in the mark recognition result storage memory 40 together with the recognition result of the mark MK (S114).

  The self-position estimation unit 42 estimates the positions of the moving body 92 and the camera 14 based on the tracking result of the feature point PT stored in the tracking result storage memory 34 by the optical flow FL, and determines the position as the first self-position. (S116). The self-position estimation unit 42 estimates the positions of the moving body 92 and the camera 14 based on the position information of the recognized mark MK and the recognition result, and estimates the position as the second self-position (S118).

  The self-position estimating unit 42 corrects the self-position by adopting either the first self-position or the second self-position based on the preset determination ranges JR1 and JR2 (S120). For example, the self-position estimation unit 42 employs the first self-position based on the optical flow FL tracking the feature point PT as the self-position SP. In this state, the self-position estimating unit 42 adopts the second self-position as the self-position SP if the second self-position based on the position information of the mark MK and the recognition result is within the determination ranges JR1 and JR2. The self position SP by one self position is reset and corrected. Thereafter, the self-position estimation unit 42 continues to estimate the self-position SP using the first self-position based on the tracking of the feature point PT with the self-position SP adopting the second self-position as a starting point. In addition to the fact that the second self-position is within the determination ranges JR1 and JR2, the self-position estimating unit 42 determines the position deviation of the first self-position, the estimated number of times of the first self-position, and the like. It may be determined whether or not the second self position is adopted as the self position SP.

  Thereafter, when there is still an image IG that is a frame image (S122: Yes), the image processing device 16 repeats Step S104 and the subsequent steps. When the image processing apparatus 16 has performed the process on all the images IG (S122: No), the self-position estimation process ends.

  As described above, in the image processing device 16, when the second self-position estimated based on the mark MK satisfies the predetermined condition, the self-position estimation unit 42 sets the second self-position as the self-position SP. Employing this, the self-position SP due to the first self-position is corrected. As a result, the image processing device 16 can improve the estimation accuracy of the self-position SP. In particular, in the image processing device 16, the self-position estimation unit 42 estimates based on the mark MK in a state where the first self-position estimated by the optical flow FL tracking the feature point PT is adopted as the self-position SP. When the second self-position satisfies a predetermined condition, the self-position SP is reset and corrected by the second self-position. As a result, the image processing device 16 can improve the estimation accuracy of the self-position SP by eliminating the misalignment by the second self-position even if the self-position misalignment due to the first self-position becomes large. Can do. In addition, when the second self-position is clearly abnormal (for example, when the second self-position is estimated to be in the air or underground when the moving body 92 is an automobile or the like), the image processing apparatus 16 Can be eliminated by a predetermined condition, so that the estimation accuracy of the self-position SP can be improved.

  FIG. 12 is a diagram illustrating another self-position estimation. In the example shown in FIG. 12, the first mark MK1 is provided on the first floor of the building 90, and the second mark MK2 is provided on the second floor of the building 90. The mark information storage memory 38 stores position information regarding the positions of the marks MK1 and MK2 with respect to the reference position P0 serving as a position reference. For example, the mark information storage memory 38 stores position information including information on the heights HD1 and HD2 from the reference position P0 to the centers of the marks MK1 and MK2. It is assumed that the image processing device 16 is provided in a movable body 92 that cannot fly such as an automobile. Accordingly, the image processing device 16 estimates the second self-position based on the captured image IG while moving in the horizontal direction.

  The self-position estimating unit 42 estimates the second self-position including the height of the moving body 92a based on the information of the height HD1 of the first mark MK1 recognized from the image IG photographed while moving in the horizontal direction. . The self-position estimation unit 42 indicates that the self-position SP of the moving body 92a is on the first floor when the height of the self-position SP falls within the first height range HA1 centered on the first reference height SH1. Information is added to the second self-position estimation result. The self-position estimating unit 42 estimates the second self-position including the height of the moving body 92b based on the information on the height HD2 of the second mark MK2. The self-position estimation unit 42 indicates that the self-position SP of the moving body 92b is on the second floor when the height of the self-position SP falls within the second height range HA2 centered on the second reference height SH2. Information is added to the second self-position estimation result. Thereby, the self-position estimation part 42 can acquire the information of the height direction which cannot be obtained by GPS (Global Positioning System), optical flow FL, etc. FIG. As described above, the self-position estimation unit 42 estimates the second self-position based on the mark MK based on the height of the self.

  When the self-position estimation unit 42 estimates the self-position SP of the moving body 92 that travels in the horizontal direction of the building 90 such as a parking lot and moves, the reference heights SH1 and SH2 with respect to the reference position P0 are marks MK1 and MK2. The second self-position can be estimated based only on the height of the self-position SP because the distance is substantially constant regardless of the distance from the head.

  FIG. 13 is a plan view seen from above for explaining the estimation of the self-positions of the movable bodies 92c and 92d that can fly. FIG. 14 is a side view for explaining the estimation of the self-positions of the movable bodies 92c and 92d that can fly. An example of the movable bodies 92c and 92d that can fly is a UAV (Unmanned Aerial Vehicle / Unmanned Air Vehicle). The marks MK3, MK4, and MK5 are provided on the wall 90a, the pillar 90b, and the like of the building 90.

  As shown in FIGS. 13 and 14, the self-position estimation unit 42 of the first moving body 92c performs the second operation based on the recognized height of the mark MK1, the distance to the mark MK1, and the deviation from the center of the mark MK1. Estimate self-position. If the estimated second self-position is within the predetermined determination range JR3, the self-position estimation unit 42 resets and corrects the self-position SP adopting the first self-position according to the second self-position. To do. Note that the predetermined determination range JR3 is defined by a rectangular parallelepiped or a cube including position information in the X direction, the Y direction, and the Z direction. If the estimated second self-position is outside the determination range JR3, the self-position estimation unit 42 estimates based on the feature points PT extracted from the images of the wall 90a, the pillar 90b, and the like without adopting the second self-position. The first self-position is used as the self-position SP.

  The self-position estimating unit 42 of the second moving body 92d estimates the second self-position based on the height of the mark MK2, the distance to the mark MK2, and the deviation from the center of the mark MK2. When the estimated second self-position is within the predetermined determination range JR4, the self-position estimation unit 42 turns and moves the second mobile body 92d based on another recognized mark MK3. May be estimated. In this case, if the self position SP is within the determination range JR4, the self position estimation unit 42 of the second moving body 92d may reset and correct the self position SP by the second self position. Thereby, the image processing apparatus 16 can further improve the estimation accuracy of the self-position SP.

  FIG. 15 is a block diagram schematically showing a hardware configuration of the image processing apparatus 16 according to the present embodiment. As shown in FIG. 15, the image processing apparatus 16 according to the present embodiment includes a CPU (Central Processing Unit) 50, a RAM (Random Access Memory) 52, a ROM (Read Only Memory) 54, and an HDD (Hard Disk Drive). ) 56, I / F 58, and bus 60. The CPU 50, RAM 52, ROM 54, HDD 56 and I / F 58 are connected via a bus 60. Further, a display unit 62 such as an LCD (Liquid Crystal Display) and an operation unit 64 are connected to the I / F 58.

  The CPU 50 is a calculation means such as a processor. The CPU 50 governs overall control of the image processing device 16. The RAM 52 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 50 processes information. The ROM 54 is a read-only nonvolatile storage medium and stores programs such as firmware. The HDD 56 is a nonvolatile storage medium capable of reading and writing information, and stores an OS (Operating System), various control programs, application programs, and the like.

  The I / F 58 connects and controls the bus 60 and various hardware and networks. The display unit 62 is a visual user interface for the user to check the state of the image processing apparatus 16. The operation unit 64 is a user interface for a user to input information to the image processing apparatus 16 such as a keyboard, a mouse, various hard buttons, and a touch panel.

  The self-position estimation program executed by the image processing apparatus 16 according to the present embodiment includes the above-described units (image acquisition unit 20, frame memory 22, image correction unit 24, image memory 26, feature point extraction unit 28, feature point storage memory). 30, a feature point tracking unit 32, a tracking result storage memory 34, a mark recognition unit 36, a mark information storage memory 38, a mark recognition result storage memory 40, and a self-position estimation unit 42). As actual hardware, the CPU 50 reads out and executes the self-position estimation program from the ROM 54, whereby the above-described units are loaded onto the main storage device. Accordingly, the image acquisition unit 20, the frame memory 22, the image correction unit 24, the image memory 26, the feature point extraction unit 28, the feature point storage memory 30, the feature point tracking unit 32, the tracking result storage memory 34, the mark recognition unit 36, A mark information storage memory 38, a mark recognition result storage memory 40, and a self-position estimation unit 42 are generated on the main storage device, and these functions are realized in a computer.

  For example, the self-position estimation program executed by the image processing device 16 of the present embodiment is provided by being incorporated in advance in the ROM 54 or the like. The self-position estimation program executed by the image processing apparatus 16 according to the present embodiment is a file in an installable or executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk). ) Or the like may be recorded and provided on a computer-readable recording medium.

  Furthermore, the self-position estimation program executed by the image processing apparatus 16 according to the present embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. good. Further, the self-position estimation program executed by the image processing apparatus 16 of the present embodiment may be provided or distributed via a network such as the Internet.

  The function, connection relationship, number, and the like of the configuration of each embodiment described above may be changed as appropriate. Each embodiment mentioned above may be combined suitably. You may change suitably the order of the step of the flowchart mentioned above.

  For example, the image acquisition unit 20 acquires an image captured by the camera 14, but may acquire a distance measurement image captured by a distance measuring device or the like.

  In the above-described embodiment, the self-position estimating unit 42 corrects the self-position SP using the first self-position based on the optical flow FL of the feature point PT by the second self-position. It is not limited to. For example, the self-position estimating unit 42 adopts the first self-position as the self-position SP when the second self-position is outside the determination ranges JR1 and JR2 while adopting the second self-position as the self-position SP. The self position SP may be corrected.

  In the above-described embodiment, the example in which the (n−1) th feature point PT is stored in the tracking result storage memory 34 and the template patch TP is generated has been described, but the (n−1) th feature point PT is previously stored. The template patch TP may be generated based on the above and stored in the tracking result storage memory 34.

  The above-described embodiments have been presented by way of example and are not intended to limit the scope of the present invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. The embodiments and modifications of the embodiments are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

16 ... Image processing device 32 ... Feature point tracking unit 36 ... Mark recognition unit 42 ... Self-position estimation unit 92 ... Moving body FL ... Optical flow HD1 ... Height HD2 ... Height IG ... Image MK ... Mark PT ... Feature point SP ... Self-position

Japanese Patent No. 4369439

Claims (9)

A feature point tracking unit that tracks feature points of the subject extracted from the image of the photographed subject;
A mark recognition unit for recognizing a mark provided in advance in the image;
A first self-position is estimated based on the tracking result of the feature points, a second self-position is estimated based on the mark, and the first self-position and the second self-position are determined based on a predetermined condition. A self-position estimation unit that employs one of
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the self-position estimation unit estimates the second self-position based on position information of the mark in which pattern information is set in advance. 3. The image according to claim 1, wherein the self-position estimation unit recognizes the mark from the image captured while moving in the horizontal direction, and estimates the second self-position based on a height of the self. Processing equipment. The image processing apparatus according to any one of claims 1 to 3, wherein the self-position estimating unit estimates the height of the second self-position based on mark information including height information of the mark. The said self-position estimation part estimates the said 2nd self-position based on the height information of the said mark, the distance to the said mark, and the shift | offset | difference from the center of the said mark. An image processing apparatus according to 1. The said self-position estimation part correct | amends the said 2nd self-position based on another mark, when the said 2nd self-position estimated from one mark is in the predetermined range. The image processing apparatus according to item 1. The self-position estimation unit adopts the second self-position when the second self-position satisfies the predetermined condition in a state where the first self-position is adopted as the self-position. The image processing apparatus according to claim 1, wherein the position is corrected by being adopted as a position. A feature point tracking step for tracking feature points of the subject extracted from the image of the photographed subject;
A mark recognition stage for recognizing a pre-set mark in the image;
A first self-position is estimated based on the tracking result of the feature points, a second self-position is estimated based on the mark, and the first self-position and the second self-position are determined based on a predetermined condition. A self-position estimation stage that employs either one of
A self-position estimation method comprising: A feature point tracking function for tracking feature points of the subject extracted from the image of the photographed subject;
A mark recognition function for recognizing a mark provided in advance in the image;
A first self-position is estimated based on the tracking result of the feature points, a second self-position is estimated based on the mark, and the first self-position and the second self-position are determined based on a predetermined condition. A self-position estimation function that adopts either one as self-position,
A program that makes a computer realize.

Images