JP5714940B2 - Moving body position measuring device - Google Patents

Description

  The present invention relates to a positioning technique for an automobile (moving body). In particular, the present invention relates to a technique for measuring the current position of a car (moving body) in real time using a car-mounted camera (its image / video) in a car-mounted information system, a microcomputer, a car navigation system, and the like. Furthermore, a technique for performing processing such as estimation / correction of the current position using position information by means such as GPS, a real image of a vehicle-mounted camera (photographed image at the time of traveling in real time), and a map DB (database) About.

  Regarding the positioning technology of the automobile according to the present invention, the background technology and its problems are as follows.

  (Conventional method 1) As a mainstream method, there is GPS + autonomous navigation. Obtains current vehicle position information (with errors) from GPS satellites, etc., and corrects the data by autonomous navigation using in-vehicle sensors (yaw rate sensors, gyro sensors, acceleration sensors, etc.) and vehicle speedometers (vehicle speed signals) Map matching is performed on a digital map such as a car navigation system (such as correcting and displaying the position of the vehicle so as not to deviate from the road). However, with this method, the single positioning accuracy is 15 meters, the output frequency is 1 Hz, and there are problems such as an output delay of 1 to 2 seconds depending on the surrounding road network conditions.

  (Conventional method 2) There is a road surface mark comparison method disclosed in Japanese Patent Laying-Open No. 2005-265494 (Patent Document 1). In this system, road marks and road signs are used as landmarks, and absolute positions (three-dimensional spatial position coordinates) of the respective landmarks are registered in a database (map DB) one by one. During real-time positioning, lane boundary lines (white lines, etc.) and landmarks are extracted from the image taken by the in-vehicle camera, and compared with the information in the map DB (consistency determination) to determine the lane position (lanes Estimate the position of the vehicle in the area).

  (Conventional method 3) There is a laser scanning method disclosed in JP-A-10-300493 (Patent Document 2). In this method, the current vehicle position is obtained by correcting the current vehicle position obtained by the navigation device based on the stationary object data detected by the radar device and the stationary object data stored in the road surrounding environment storage device. The position error can be made from about several tens of meters to about 10 cm to several tens of centimeters, and the accuracy can be improved.

  (Conventional method 4) Further, there is a high-precision RTK (Real Time Kinematic) -GPS method. With this method, positioning accuracy with an error of about 2 to 3 cm can be obtained. However, positioning devices using the conventional methods 3 and 4 are very expensive, and are not good at performing positioning on high-speed moving bodies in real time, for example.

  (Conventional method 5) There is also a hybrid method proposed by the present inventor (Patent Document 3). In this method, in addition to the existing car navigation system, a three-dimensional inertial gyroscope sensor, a vehicle speed sensor, and a white line detection result are integrated to estimate an accurate vehicle position in real time. This technique achieves a real-time positioning speed, but the average positioning accuracy is only 3 to 5 meters.

  (Conventional method 6) Further, as a positioning method using an in-vehicle camera, there is JP 2010-151658 A (Patent Document 4) ("mobile body position measuring device" etc.) proposed by the present inventor. In this method, features (feature information) extracted from real images (road surface and landscape in the traveling direction) of a camera (imaging unit) mounted on a moving body (own vehicle) and features included in a map DB (map information) The position of the moving body is measured (estimated) by performing a matching process on the part (feature information). The present invention and Patent Document 4 share a large framework, but have different technical features.

  In Patent Document 4, specifically, a white line (lane boundary line) on the road surface in the traveling direction of the host vehicle is imaged and recognized, and an offset distance from the white line of the moving body is detected to correct the position of the moving body. The accuracy of the vehicle position is increased by processing such as selecting an appropriate lane from a plurality of lanes. In addition, at a reference point such as an intersection provided at a predetermined interval, it is possible to store an image captured from a plurality of positions facing the road direction in association with the point (reduction of processing due to unnecessary three-dimensional image). ). Further, by storing only the feature data (feature information) instead of the three-dimensional image (actual image data) in advance, it is possible to increase the speed by omitting the extraction process and to reduce the necessary storage capacity. Also, assuming multiple locations where the vehicle can exist, such as when there are multiple lanes based on map information, for example, from the possible locations (lanes), for example, information on feature parts of the front image The difference is calculated by comparing with the information of the feature portion of the real image extracted by the real-time processing (matching processing).

  On the other hand, there is a technique called SFM (Structure from Motion) related to a positioning technique using a camera image. For example, it is described in Non-Patent Documents 1 and 2. In the SFM method, in a robot (moving body) equipped with a camera, a relative position / posture before and after the movement of the robot is estimated simultaneously with restoration of a three-dimensional environment from only image information captured by the camera.

  Further, image analysis processing for extracting feature amounts (feature portions, feature information) from an image is described in Non-Patent Document 3, for example.

JP 2005-265494 A (Road surface mark comparison) JP-A-10-300493 (Laser scan) JP 2008-175786 A (Hybrid) JP 2010-151658 A (Camera use)

C. Tomasi and T. Kanade, “Shape and motion from image streams under orthography-A factorization method,” Int. J. Comput. Vision, 9-2, pp. 137-154, Nov. 1992. Takeo Kanade, Conrad Poleman, Toshihiko Morita, “Reconstruction of object shape and camera motion by factorization method,” IEICE Transactions D-II, J74-D-II-8, pp. 1497-1505, Aug. 1993. David G. Lowe, "Distinctive image features from scale-invariant keypoints", International Journal of Computer Vision, 60, 2 (2004), pp. 91-110.

  As a problem, for example, in the positioning technique using an in-vehicle camera as in the conventional method 6 (Patent Document 4), it is inexpensive (does not use an expensive device) and has high accuracy (at least higher than a general means such as GPS). I want to achieve accurate positioning.

  However, the positioning technique like the conventional method 6 (Patent Document 4) is not a technique for realizing complete real-time positioning, but a positioning technique using virtual image processing (three-dimensional CG processing). For example, Patent Document 4 is a technique that uses an automatic generation function of a virtual landscape image for 3D navigation, and generates each virtual landscape image (3D CG) from a plurality of possible vehicle existence positions (lanes) on the road.・ Acquire and perform a process of extracting a feature from the virtual landscape image and the actual image taken at the present time, and determine an optimum position by a feature matching process (for example, paragraph [0004] of Patent Document 4) , [0006], [0032], etc.).

  However, in the positioning technique using the virtual image processing, for example, a large cost is required for producing and maintaining a city map DB for three-dimensional CG navigation. Moreover, it cannot be applied to all roads recorded in the map DB, and it cannot flexibly cope with changes in road scenery such as new construction.

  Therefore, as a problem, it is desired to realize real-time positioning not using (or not requiring / reducing) the virtual image processing, and to make the production and maintenance of the map DB more efficient and lower cost. In addition, we want to be able to adapt to many roads recorded in the map DB and flexibly respond to changes in the road landscape.

  When performing real-time positioning without virtual image processing, it is necessary to appropriately extract and determine the features (feature information) extracted from the real image of the camera (roadscape) so that they can be used in the positioning calculation. Come. However, there is a problem that the appropriate extraction and determination are difficult because the content (landscape) of the actual image changes greatly due to factors such as road surrounding conditions and time passage during positioning (when imaging). The above factors include, for example, driving time (season), weather and lighting conditions, road congestion (including vehicles and pedestrians), and building environment (including roads, roadside buildings, trees, signs, etc.) Fluctuations. For example, if a characteristic portion corresponding to a car or a tree included in a landscape (actual image) is detected and used in positioning calculation, the processing speed and positioning accuracy are lowered.

  As described above, the main object of the present invention relates to a real-time positioning technique using an in-vehicle camera, and is to provide a technique capable of realizing a low-cost and high-accuracy positioning, particularly using virtual image processing. Robust features that can realize real-time positioning, improve the efficiency of production and maintenance of the map DB, reduce costs, and adapt to factors and fluctuations such as road surroundings during positioning (when imaging) It is to provide positioning technology including methods such as detection.

  In order to achieve the above object, a typical embodiment of the present invention is a mobile body position measuring apparatus related to a mobile body such as an automobile and an information system mounted thereon, and has the following configuration. And

  This apparatus is a mobile body position measuring apparatus that measures or estimates the position of a mobile body, and (a) the current approximate position and orientation of the mobile body using means such as GPS in real time when the mobile body is traveling And (b) photographing at least one first direction (for example, forward) around the moving body in real time when the moving body is mounted on the moving body. One or more imaging units (cameras) for obtaining a first image (actual image group), and (c) a plurality of actual images photographed in the vicinity of a reference point (RP) on a map as the first image. A first feature extraction unit that extracts a feature portion including a straight line or a corner point (feature point) from the image and sets the feature portion as first feature information; and (d) a first feature near a predetermined reference point on the map. A feature part including a straight line and a corner point is extracted from the two images (actual image group) as second feature information, A map database in which data including the feature information of 2 is registered in advance, and (e) when the mobile object travels, from the map database based on the first position information, corresponding to the vicinity of the reference point A second feature extraction unit that reads out and acquires the second feature information to be acquired, and (f) inputs the first feature information and the second feature information, and compares and matches the respective feature units. Thus, the positions of the plurality of feature portions in the vicinity of the reference point are estimated, and based on the positions of the plurality of feature portions, the position of the moving body at each time point associated with the movement is A feature matching unit that calculates a relative positional relationship with the position of each of the plurality of feature units to estimate the current position of the moving body and outputs the second position information; and (g) the first The second position with respect to the position information of one By correcting using a broadcast, having a position correction section for outputting a current position of the moving body as the third position information.

  Furthermore, the present apparatus is related to the feature extraction process and has a robust feature detection unit. The feature detection unit inputs real image data captured by the camera, performs processing in the region division unit, the feature extraction unit, and the unnecessary object removal unit, and outputs feature information as a result. The area dividing unit includes a road area detecting unit and a roadside area detecting unit. The unnecessary object removing unit includes a road unnecessary material removing unit and a road side unnecessary material removing unit.

  The road area detection unit detects the road area (A) from the input real image area. For example, the road area detection unit detects a road surface mark such as a white line from the area of the actual image, and estimates and determines the road area (A) including the lane based on the shape of the detected road surface mark. The roadside area detection unit detects the roadside area (B) from the input real image area. For example, the roadside area detection unit estimates and determines the roadside area (B) from the real image area based on the shape of the road area (A) detected by the road area detection unit. The area dividing unit outputs information on the road area (A), the roadside area (B), and the other area (C) in the real image area.

  The feature extraction unit obtains a feature group (corner points, line segments, color regions, etc.) from each region (road region (A), roadside region (B), other region (C)) in the real image by image analysis processing. (Feature part) is extracted.

  The road unnecessary object removal unit detects a dense horizontal line area (an area where a plurality of horizontal lines are gathered at a high density) with respect to the road area (A) of the actual image, and deletes a feature group corresponding to the detected area. . The roadside unnecessary object removal unit detects a dense corner point region (a region where a plurality of corner points are gathered at high density) with respect to the roadside region (B) of the actual image, and a feature group corresponding to the detected region Is deleted. The unnecessary object removing unit outputs the feature information (feature portion group information) after the removal (deletion).

  According to the representative embodiment of the present invention, it is related to the technique of real-time positioning using an in-vehicle camera, and low-cost and high-precision positioning can be realized. In particular, real-time positioning that does not use virtual image processing can be realized, map DB production and maintenance can be made more efficient and less expensive, and it can be applied to factors and fluctuations such as road surroundings during positioning (when imaging) It is possible to provide a positioning technique including a technique such as robust feature detection.

It is a figure which shows the structural example of the motor vehicle which is the system (mobile body position measuring apparatus) of one embodiment of this invention, and a vehicle-mounted information system. It is a figure which shows typically an example, such as a characteristic part of RP vicinity on a map in a certain time and position. It is a figure which shows the example of the feature point etc. in the real image imaged at a certain time and position. It is a figure which shows the example of the relative relationship of the own vehicle position and the feature point position between the real images of the time and position before and after movement. It is a figure which shows the detailed process example of a prior registration process. It is a figure which shows the detailed process example of a real-time positioning process. It is a figure which shows roughly about the position estimation etc. which used the SFM method. It is a figure which shows the structural example of the feature detection part which implemented the robust feature detection method as detailed embodiment. It is a figure which shows the example of each area | region and thing on a road. It is a figure which shows the example before the area | region division and unnecessary object removal in a real image. It is a figure which shows the example after the area | region division and unnecessary object removal in a real image. It is a figure which shows the example of the limitation process in a real image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted. Abbreviated as RP: Reference Point, etc. As symbols for explanation, T: time point, K: own vehicle position (≈camera imaging position), G: image / real image, P: feature point (and feature point position), and the like.

  As a main feature of this embodiment, when performing real-time positioning of a moving body (automobile), feature matching processing using the SFM method is performed (FIG. 1, FIG. 5, FIG. 6 etc.). It has a robust feature detection method (FIG. 8, etc.) that can adapt to fluctuations. Thereby, the current position of the host vehicle is estimated with low cost and high accuracy. In particular, as shown in FIG. 8, it has a processing function for detecting various regions including roads and roadsides from a roadscape (actual image) and removing unnecessary objects (corresponding features) that adversely affect positioning.

[system]
FIG. 1 shows a configuration example of an automobile and an in-vehicle information system 100 that are systems (moving body position measuring devices) according to an embodiment of the present invention. The automobile and in-vehicle information system 100 includes a real-time positioning unit 1, a control unit 2, sensors 3, a camera (imaging unit) 4, a map DB (database) 5, and the like. The control unit 2 includes, for example, a CPU, a ROM, a RAM, and the like, and realizes control of the entire system and processing functions including the real-time positioning unit 1 by processing of circuits and programs.

  In addition, although this user (100) uses a user, in the case of a measurement vehicle to be described later, a pre-registration processing unit corresponding to the real-time positioning unit 1 is provided, and processing (d2) for pre-registration (a2) to the map DB 5 Registration process). Moreover, the form (form which updates map DB5 content (d2) at any time) which mounts a prior registration process part in a user's motor vehicle (100) is also possible.

  The sensors 3 are known elements such as the GPS unit 31, the gyro sensor 32, and the travel sensor 33 (may have other means). The GPS unit 31 receives carrier waves (for example, once per second) from a plurality of GPS satellites, and detects the position based on the latitude and longitude from the arrival time difference. The gyro sensor 32 detects the azimuth angle (orientation) of the host vehicle from the change in the angular velocity of the host vehicle. The travel sensor 33 (such as a vehicle speedometer) detects the travel distance and speed of the host vehicle.

  One or more cameras 4 are mounted on one automobile (100). Various forms are possible for the mounting position and orientation of the camera 4 and the detailed functions. For example, as shown in FIG. 3, the vehicle is mounted at a position / orientation for imaging a road surface and a frame including a landscape in front of the automobile (100). During real-time positioning, the camera 4 captures a road, a landscape, and the like in the traveling direction (corresponding to the direction of the camera 4) as the vehicle moves, and obtains a plurality of real image frames. The actual image data may be temporarily stored in a storage unit as appropriate. The camera 4 can be an inexpensive monocular camera such as a rear view camera or a drive / recorder camera.

  When two or more (a plurality of) cameras 4 are mounted on one automobile, a plurality of real images simultaneously captured by the plurality of cameras 4 are used for feature extraction processing (image analysis processing) or the like. Thereby, the positioning accuracy can be increased instead of the cost. With respect to the plurality of real images, for example, a shape in which a landscape in a plurality of directions is imaged from the same imaging position (camera position) or a shape in which the same object is imaged from a plurality of imaging positions (camera positions) is possible. is there.

  The map DB 5 stores data such as map information (d0), RP information (d1), and second feature information (d2). The map DB 5 is realized by storage means (for example, HDD, DVD, etc.) not shown, a service on a communication network (the contents of the map DB 5 are automatically updated), and the like. In the process of pre-registration (a2) in the map DB 5, a feature portion is extracted from an actual image near a predetermined RP, and is registered in association with RP information (d1), second feature information (d2), and the like. The predetermined RP includes at least a road intersection.

  The map information (d0) includes known road information, landscape information, and the like. In this example, data used in the digital map of the car navigation 6 is assumed. Further, as elements relating to the present invention, the map information (d0) particularly includes information (d1) relating to a predetermined RP, and the RP information (d1) is second feature information (d2) associated with the RP. including.

  The RP information (d1) is information on an RP such as an intersection to be referred to in real-time positioning on the map, landmarks, and the like. Identification information and attribute information about the RP etc., three-dimensional space (map) Information such as position coordinates (three-dimensional coordinates {X, Y, Z} or information of a predetermined format such as corresponding latitude / longitude).

  The second feature information (d2) is used for pre-registration (a2) regarding landmarks such as signs and buildings included in the scenery (actual image) near the RP identified by the RP information (d1). Information such as feature quantities and position coordinates (two-dimensional coordinates {x, y} and corresponding three-dimensional coordinates {X, Y, Z} in the image) relating to feature parts (feature points (P), etc.) extracted by processing including.

  The car navigation 6 is a car navigation system, a display, or the like that uses and outputs information on the current position of the vehicle by the real-time positioning unit 1. The car navigation 6 uses the data in the map DB 5, the third position information (L3), and the like to generate data such as a navigation map as needed, and performs screen display processing and the like. Further, as described above, the vehicle current position and azimuth information may be transferred to a microcomputer or ECU for vehicle control.

[Real-time positioning unit, processing overview]
With reference to FIG. 1, the configuration of the real-time positioning unit 1 in the present embodiment (automobile and in-vehicle information system 100) and an outline of the processing at the time of the real-time positioning and the like will be described in (0) to (5) below. . The real-time positioning unit 1 performs processing related to real-time positioning by the control unit 2 in cooperation with the sensors 3, the camera 4, the map DB 5, the car navigation 6, and the like. The real-time positioning unit 1 includes a position detection unit 10, a first feature extraction unit 11, a second feature extraction unit 12, a feature matching unit 13, a position correction unit 14, and the like. Other known elements such as an image data storage unit and a map matching unit may be provided as necessary. In addition, about the process of each part at the time of a user using this system with a general vehicle, it is a process in real time (real time), and pre-registration (a2) about data {d0, d1, d2} of map DB5 ) (For example, processing by the system provider or measurement vehicle).

  (0) The position detection unit 10 inputs the detection information (b0) from the sensors 3 including the GPS unit 31 and the like in real time as needed, and uses the data to approximate the current position and azimuth of the host vehicle. Etc. (schematic first position having a positioning error by general GPS) is detected, and this detection information is set as first position information (L1). For example, the situation is shown in FIG.

  (1) The first feature extraction unit 11 obtains data (b1) of an actual image (a plurality of image frames on the time axis for each camera 4) captured by the camera 4 by an input at the actual time (a1). It inputs, performs the process which extracts the characteristic part regarding the scenery containing the road of the surrounding environment of the own vehicle from the real image (for example, FIG. 3), and outputs it as 1st characteristic information (c1). The feature part and the first feature information (c1) are the coordinates ({x, y}) of the corner point (feature point (P)) included in the real image (two-dimensional landscape image) and the corner point. Includes various types of information that can be extracted by predetermined image analysis processing, such as feature quantities (eg, SIFT feature quantities; see, for example, Non-Patent Document 3) representing features such as shape, color, and brightness change (described later, such as FIG. 3). . Real-time positioning is started at least from a position near the RP (such as the front) such as an intersection. The first feature extraction unit 11 performs the feature extraction processing (processing of a plurality of real image frames on the time axis) from a position near the RP within the visual field range according to the first position information (L1). .

  (2) The second feature extraction unit 12 performs pre-registration (a2) based on the first position information (L1) accompanying the movement of the host vehicle in parallel with the operation of the first feature extraction unit 11. ) From the registered map DB 5 in which each data (d0, d1, d2) is registered, map information (d0), RP information (d1) and second feature information (d2) in the vicinity of the first position of the vehicle. Are acquired sequentially. The second feature extraction unit 12 reads out and inputs the data (b2) from the map DB 5, extracts the feature from the landscape (actual image) in the vicinity of the corresponding RP, and first feature information (c1) Is output as second feature information (c2) for matching with (reference source). The data (b2) includes second feature information (d2) corresponding to information on a feature portion near the corresponding RP. If the corresponding second feature information (d2) is pre-registered (a2) in the map DB 5, it is only necessary to read and acquire the data. The second feature information (c2) has the same format as the first feature information (c1), and includes information such as feature amounts and position coordinates regarding feature points.

  In each feature extraction process in the real time (a1) and pre-registration (a2) in (1) and (2) above, roads and scenery near the RP (for example, the intersection in FIG. 2) on the map are targeted. Image analysis processing is performed to extract features (feature points, etc.) relating to landmarks (white lines on road surfaces, signs, buildings, etc.) from the obtained images (described later, FIGS. 5 and 6).

  (3) The feature matching unit 13 inputs the first feature information (c1) and the second feature information (c2), and performs the feature matching process using the SFM method, so that the current vehicle position is sequentially And the result (estimated current position of the vehicle) is output as the second position information (L2). Relative positional relationship with a plurality of feature parts (feature points (P), etc.) in a frame of a plurality of real images (G) at a plurality of time points (T) and positions (K) accompanying the movement of the host vehicle The current position (K) of the host vehicle is estimated by calculating (relative position / orientation of the host vehicle) using the SFM method (described later, such as FIG. 4).

  (4) The position correction unit 14 inputs the first position information (L1) and the second position information (L2), corrects the current position of the vehicle using L2 with respect to L1, and obtains the result. Output as third position information (L3). L3 includes information on the current vehicle position and azimuth angle.

  The second position information (L2) may be output as the position information of the final result without providing the position correction unit 14. At least L2 can be obtained with higher accuracy than L1. It is preferable to perform correction by the position correction unit 14 at a place other than the vicinity of the RP.

  When performing map matching processing or the like, for example, correction is performed so that the vehicle does not come off the road in the map based on the first position information (L1) or the like. For example, a position where the vehicle can exist is selected from L1 and map information (d0) in the vicinity thereof. Further, based on L1, map information (d0) such as the number of lanes and the lane width of a road, a mark on a road surface, a sign / signal, and the like may be acquired to grasp a possible position of the vehicle.

  (5) In the vehicle and in-vehicle information system 100, processing is performed using the third position information (L3). For example, the current position of the vehicle is displayed on the car navigation 6 using L3 or the like. Moreover, it is good also as a form etc. which transfer and utilize the information of the vehicle present position and azimuth angle by said L3 to the microcomputer and ECU for vehicle control.

[Pre-registration process]
The process at the time of prior registration (a2) to the map DB 5 is performed by, for example, a measurement vehicle (including a camera and a prior registration processing unit). The pre-registration processing unit includes a processing unit similar to the first feature extraction unit 11 and the like. The vehicle is actually driven on the target road by the measurement vehicle, and the landscape near the target RP is imaged. The actual image data is temporarily stored, and the feature is extracted from the actual image data in the same manner as the process in the first feature extraction unit 11, and this result is used as the second feature information (d2). Register in the DB 5 in association with the RP information (d1). This processing is generally the same as the processing content in the first feature extraction unit 11 at the time of real-time positioning. In this process, unlike real-time positioning, it is not necessary to process everything in real time. For example, the feature extraction processing may be executed by collecting actual image data collected and accumulated in advance.

[RP, landmark]
In FIG. 2, examples of maps, roads, RPs, landmarks, feature points (P), and the like are schematically shown by a two-dimensional plane (note that they are for explanation purposes and are different from the map displayed on the car navigation system 6). Examples of own vehicle 200, intersection 201, road 202, road / lane 203, white line (lane boundary line) 204, building 205, roadside 206, and the like are shown. K indicates the current position of the host vehicle 200 and is expressed by three-dimensional coordinates {X, Y, Z}, an azimuth (arrow), and the like.

  RP is a positioning reference point having a landscape feature useful for positioning, and indicates a road intersection such as 201, a branching / merging portion, and a landmark point. As the RP, the main thing (useful for positioning) in the reality (map) is pre-registered (a2) as the RP information (d1). The RP is not limited to the intersection 201, and may be a plurality of points at predetermined intervals, a building 205 serving as a landmark, or the like. The three-dimensional position information of the RP is calculated by the measurement vehicle when the map DB 5 is constructed (a2) and registered in the RP information (d1). The calculation process of the three-dimensional position information of the RP is described separately (described in the pre-registration process in FIG. 5). Although not registered in the present embodiment, real image data in the vicinity of the RP captured in advance may be registered in association with each other.

  FIG. 2 shows the current position (K) of the host vehicle 200 when the host vehicle 200 is about to enter the RP on the road 202 / lane 203 near the intersection 201 (RP). As information of the position (K) at a certain time (T), the three-dimensional position coordinates ({X, Y, Z}) and azimuth of the RP registered in advance in the map, and an error range (described later) of the position data Information). Note that not only coordinates but also azimuth angles are abbreviated as position information. A circle indicates a feature point (P) as an example of a feature portion extracted from a real image with respect to a landmark or the like. For example, a corner point of a white line 204 on the road surface in the traveling direction of the host vehicle 200 or both sides of the traveling direction. The corners of the building 205 are shown. The broken line indicates the distance (relative positional relationship) between the current position (K) of the host vehicle 200 (≈the imaging position of the camera 4) and each of the plurality of feature points (P). The position of the feature point (P) is indicated by the coordinates ({X, Y, Z}) on the map in FIG. 2, but information on the coordinates ({x, y}) of the corresponding pixel in the actual image. Etc.

[Real image, feature extraction processing]
FIG. 3 schematically shows an example of an actual image (G) captured by the camera 4 at a certain time (T) and position (K). It is a case where the front is imaged from before the intersection (RP). Circles are examples of feature points (P). The presence of other vehicles is omitted for clarity. Reference numeral 301 denotes a road area (running lane area), and 302 denotes a white line (white line area). Each feature point (P) has pixel position coordinate information ({x, y}) in the real image.

  A real image (target for which a feature part is to be extracted) for pre-registering the second feature information (d2) in the map DB 5 is captured from the front of the RP, for example, in front of the RP as shown in FIG. An image. In the case of the intersection 201 in FIG. 2, each image from four directions entering the RP is used.

  In the feature extraction processing for extracting feature points (P) and the like, basically, various known techniques (image analysis processing) can be applied. In the image analysis processing, for example, a line segment or corner corresponding to the white line (204, 302) on the road surface, the outline of the building (205), or the like is calculated by calculating a difference value such as brightness and luminance of pixels constituting the actual image. It is possible to extract a feature amount that represents the shape, color, brightness change, etc. of a point. Note that the feature to be extracted is not limited to a line segment or a corner point, but may be a predetermined figure (such as an arc), depending on the processing technique to be applied.

[Relationship between own vehicle and feature points, feature matching processing]
FIG. 4 shows an example of the relative positional relationship between the vehicle position (K) and the feature point (P) due to the movement of the vehicle. When there are a plurality of feature points (P) in each real image (G) at a plurality of time points (T) and positions (K) accompanying the movement of the vehicle, the current vehicle position (K) is estimated by the SFM method. To do. For example, consider two time points (T1, T2) before and after the movement of the vehicle. Each corresponding position (K1, K2) and real image (G1, G2) are included. Corresponding to each, there is a relative distance between the position of a plurality of feature points P (for example, P1 to P4) and the position (K) of the host vehicle. 401 (broken line) is a distance corresponding to T1 and the like, and 402 (solid line) is a distance corresponding to T2 and the like.

  The characteristic portion changes depending on the position (K) when the real image (G) is captured. In other words, the information on the characteristic part includes information on the position (K) at the time of imaging. Accordingly, the current vehicle position (K) can be estimated by calculation using the SFM method.

  In the feature matching process, for example, a plurality of feature points P (first feature information (c1)) extracted at the time of real-time positioning and a plurality of feature points P (first items) read out from the map DB 5 correspondingly. 2 feature information (c2)), the positions of a plurality of feature points P are compared, and the degree of coincidence / similarity is determined. For example, as a frequently used technique, the Euclidean distance is calculated, and the one with the smallest distance is set as a matching feature point.

[Location estimation using SFM method]
FIG. 7 schematically shows position estimation (feature extraction / feature matching processing) using the SFM method. This content corresponds to the content of feature extraction processing and feature matching processing in the real-time positioning unit 1 and the pre-registration processing unit.

  The feature point (P) is extracted from a two-dimensional image (for example, FIG. 3), and based on the two-dimensional coordinates ({x, y}), the three-dimensional spatial position ({X , Y, Z}) (for example, FIG. 2). In the pre-registration (a2) process, a feature amount (for example, SIFT feature amount) representing a feature such as a shape, color, or brightness change at the feature point (P), an estimated P position, and its estimation accuracy (both errors are shared). Information such as (dispersion matrix) is registered in the map DB 5 (d2) (described later, FIG. 5). On the other hand, in the real-time positioning process, the feature amount (c1) of each feature point (P) extracted in the processing of the real time (a1) and each feature point (P) registered in the map DB 5 (d2). Matching with the feature amount (c2) is performed. When this association is possible, the present vehicle position (K) is estimated by the SFM method using this result (each P arrangement) (for example, FIG. 4), and the estimation accuracy (covariance matrix of error) is also obtained. It can be calculated at the same time.

[Details]
Based on the above, a detailed processing example in the present embodiment will be described below. In this system (real-time positioning unit 1), by utilizing the SFM technique, matching of the characteristic parts (c1, c2) of a plurality of real images (G) taken from the vehicle-mounted camera 4 is performed with high-precision three-dimensional relative The position (K) is determined by estimation calculation.

  As a process of pre-registration (a2) of the feature quantity (second feature information (c2)) as a part of this method, a plurality of real images (G that have been captured in the vicinity of the RP such as the intersection (201) ) To extract a characteristic portion (such as a characteristic amount and three-dimensional position coordinates) such as a characteristic point P, and estimate the vehicle position (K). The feature amount in the vicinity of the RP is optimized, and the optimum feature amount is sequentially selected until the measurement error (estimation accuracy) of the vehicle position (K) is suppressed within a predetermined range. The optimum feature quantity is selected so as to minimize the measurement error (estimation accuracy) of the vehicle position (K), and at the same time, the three-dimensional spatial position of the feature quantity and the estimation accuracy can be obtained. Information on the finally selected feature is registered in the map DB 5 as second feature information (c2).

  On the other hand, in the real-time positioning unit 1, information (first feature information (c1)) such as feature amounts extracted from the real image (b1) of the camera 4 and information read from the corresponding map DB 5 (first) The feature matching process is performed with the feature information (c2)), and the relative position and direction between the current position (K) of the vehicle and the features (plural feature points P) near the RP are measured by the SFM method. Thus, highly accurate positioning of the current vehicle position (K) is realized. The real-time positioning unit 1 performs the above feature matching process and the like from at least a position near the RP such as an intersection with the movement of the own vehicle, and processes the real image (G) at each time point (T). Sequentially, the positioning error of the current vehicle position (K) is minimized, and the positioning result is output when it reaches a predetermined allowable accuracy (range).

  In this system, in the process using the SFM method, the feature amount is optimized based on the principle of the known extended Kalman filter (a process of selecting an optimum feature from a plurality of real images taken continuously). At the same time, three-dimensional positioning (calculation of three-dimensional position coordinates) is performed. This processing can be divided into pre-registration processing (FIG. 5) and real-time positioning processing (FIG. 6). As a difference, for example, each feature point position is estimated in FIG. 5, but is not estimated because it is registered in FIG.

[Pre-registration process (details)]
In the process at the time of pre-registration (a2), in order to create the map DB 5 including the information (d2) of the characteristic part in the real image near the predetermined RP, the measurement vehicle (pre-registration processing unit) Measurement starts from the position before RP. For example, the RP / traveling direction is imaged from one or more cameras mounted in front of the measurement vehicle, and feature extraction processing (image analysis processing) is performed on a plurality of real images on the time axis.

  As a detailed configuration / processing example in the pre-registration processing unit, processing using the SFM method and the like will be described with reference to FIG. Reference numeral 500 denotes data information such as an input actual image.

  (1) First, in the road area detection process 501, a road area (and a non-road area) is detected from the input image (500). The road area (traveling lane area or the like) is, for example, an area such as 203 in FIG. 2 or 301 in FIG. 3, and is an area partitioned by white lines or the like as 204 in FIG. 2 or 302 in FIG. Details of this processing 501 will be described later.

  (2) In the unnecessary object removal process 502, a predetermined process is performed on the result of (1) above to remove the feature points of the unnecessary object (feature part group corresponding to the variable object that adversely affects the positioning calculation). Details of this processing 502 will be described later.

  Through the above (1) and (2), a characteristic portion such as a plurality of characteristic points (P) related to the actual image (G) at each time point (T) and position (K) (FIGS. The feature amount and position coordinate information regarding 4) are obtained as feature information.

  (3) In the measurement process 503 (SFM process), using the extended Kalman filter (prediction formula and initial value) with the result of (2) as an object (input), the three-dimensional space of each feature point (P) While estimating the position ({X, Y, Z}) and the current vehicle position (K), the respective estimation accuracy (covariance matrix of error) is calculated. As shown in FIG. 5, (a) the current position (K) of the vehicle, (b) the position of each feature point (P), and (c) the estimation accuracy (covariance matrix of errors) of each of them, etc. Perform estimation processing. The movement (variation amount) is used for the vehicle position of a. A detailed calculation formula will be described later.

  (4) Initial value setting 504: As the initial value of (3) above, a value preset in this system is input. The initial values are the initial feature point position, the initial vehicle position (motion), and the like. As a method of setting the initial value, for example, an empirical value is used, or a rough position is set from the map information (d0) of the map DB 5. For example, when two cameras 4 are used, it can be determined and set from depth information by stereo calculation.

  (5) Sequential variation input 505: For the prediction formula of (3) above, the vehicle speed / direction change by the sensors 3, etc. until the frame of the real image (G) at the next time (T) is captured. Are sequentially input.

  (6) In the determination process 506, based on the estimation results (a to c) in (3) above, the estimation accuracy (error) of c at the time (T), position (K), and actual image (G). For example, it is determined whether or not the value of the covariance matrix is within a predetermined range (below the threshold). When the c value is out of the range, the process returns to the beginning, and subsequently, the frame of the real image (G) at the next time point (T) is input, and the same calculation process (501 to 503) is repeated. When the c value falls within the range, the estimated value (ac) is selected as the optimum value at that time, and the measurement process is terminated.

  In the process of (6) above, while the estimation accuracy (c) of the vehicle position (K) is suppressed within a predetermined range, a frame of the next real image (G) is input and a new feature amount is introduced. The feature amount that could not be tracked between the previous and next frames is deleted. A new feature amount can be selected and introduced from the estimation accuracy (c) of the vehicle position (K). For example, when the accuracy (c) of the position (K) in the vehicle traveling direction (for example, the Y direction in FIG. 2) is low, a new feature point at a distant place can be selected, and the lateral direction with respect to the vehicle traveling direction When the accuracy (c) of the position (K) in the (for example, the X direction in FIG. 2) is low, a new feature point in the vicinity can be selected.

  (7) In the output process 507, the information (including the list of optimum features) of the selection results (a to c) in (6) above is used to estimate the vehicle position (K) with respect to the RP position ( Corresponding position on the map) and feature information (c2). And in the case of prior registration (a2), the said output information is registered into map DB5.

[Real-time positioning process (details)]
In the processing at the time of real-time positioning, the RP / traveling direction is imaged from the camera 4 of the general vehicle (100), and the map DB5 (RP information (d1)) is used for the camera 4 based on the first position information (L1). The RP within the visual field range is searched, and the real-time positioning process is started from the position before the RP.

  As a detailed configuration / processing example in the real-time positioning unit 1, processing using the SFM method and the like will be described with reference to FIG. 6 mainly corresponds to the processing of the first feature extraction unit 11 and the feature matching unit 13 in FIG. Reference numeral 600 denotes input real image data (b1). Hereinafter, the processing content overlapping with FIG. 5 will be described briefly.

  (1) First, in the road area detection processing 601, a road area (and a non-road area) is detected from the input image (600) (similar to 501). Details of this processing 601 will be described later.

  (2) In the unnecessary object removal process 602, a predetermined process (similar to 502) for removing the feature point (P) of the unnecessary object is performed on the result of (1). Details of this processing 602 will be described later.

  Through the above (1) and (2), a characteristic portion such as a plurality of characteristic points (P) related to the real image (G) at each time point (T) and position (K) (FIGS. The feature amount and position coordinate information regarding 4) are obtained as feature information.

  (3) In the measurement process 603 (SFM process), using the extended Kalman filter (prediction formula and initial value) with the result of (2) as an object (input), the three-dimensional space of each feature point (P) Each estimation accuracy (covariance matrix of error) is calculated while estimating the position and the current vehicle position (K). As shown in FIG. 6, calculation processing is performed to estimate (a) the current position (K) of the host vehicle, (c) the accuracy of estimation of the current position (covariance matrix of errors), and the like. A detailed calculation formula will be described later.

  (4) Initial value setting 604: The initial value of (3) above is obtained from the second feature information (c2) registered in association with the first position information (L1) and the map information (d0) of the map DB5. The target position of each feature point (P) is estimated, and the initial value of (3) above is set.

  (5) Sequential variation input 605: For the prediction formula of (3) above, the vehicle speed / direction change by the sensors 3, etc. until the frame of the real image (G) at the next time point (T) is captured. Are sequentially input.

  (6) In the determination process 606, based on the results (a, c) of (3) above, the estimation accuracy of c (coincidence of errors) at the time (T), position (K), and actual image (G). It is determined whether or not the value of (dispersion matrix) is within a predetermined range (below the threshold). When the c value is out of the range, the process returns to the beginning, and the next frame of the real image (G) is input, and the same calculation process (601 to 603) is repeated. When the c value falls within the range, at that time, the estimated value (a, c) is selected and the process ends.

  (7) In the output process 607, the information of the selection result (a, c) in (6) is output as an estimated value (corresponding to L2) of the current vehicle position (K).

[a formula]
Calculation formulas and the like in the measurement process (SFM process) of (3) are as follows.

  The prediction formula of the extended Kalman filter is the following formula (1).

  Here, the state vector X and the measurement error covariance matrix (prediction error) Q are defined by the following equation (2).

  X is a joint vector of the vehicle state vector V representing the relative position / orientation of the vehicle (camera) to be measured and the two-dimensional coordinate vector P of the feature point of the landscape image. It comprises a quantity R and a translation quantity T) and 2N elements (N: number of feature points).

  In the sequential loop of the Kalman filter, the following equation (3) is satisfied.

  Here, Gv is a Jacobian matrix of the vehicle state (position and orientation), Gp is a Jacobian matrix of feature points, and Gv and Gp are the following equations (4) and (5).

  In the Kalman filter update process, the following expression (6) is satisfied.

  Here, the above u and v are the following equations (7).

prediction point: image projection position of predicted feature point, matched point: two-dimensional image detection position of matched feature point, Mc: camera internal matrix, R _{α, β, γ} : camera rotation matrix, P: three-dimensional feature point Absolute position, V: Indicates the current position of the vehicle.

[Robust feature detection method]
Next, a robust feature detection method that can be adapted to changes in roadscape will be described as a detailed embodiment based on the above configuration with reference to FIGS. In this configuration, a feature part (group) is appropriately extracted from a real image (for example, FIG. 3) of a fluctuating road landscape, and effective positioning calculation (SFM processing in the feature matching unit 13 (measurement processing in FIGS. 5 and 6). 503, 603)), a process (21) for detecting (dividing) each area including the road area and the roadside area from the actual image, and a process (23) for removing "unnecessary items" from each area. I do.

  “Unnecessary material” is an entity (corresponding information) whose features in the landscape (actual image) fluctuate greatly due to factors such as fluctuations in the environment around the road and the passage of time. Feature information used for positioning calculation As such, it is not useful information. For purposes other than the positioning calculation (for example, vehicle control for safety), even the “unnecessary items” can be useful information and may be detected and used.

  FIG. 8 shows a configuration example of the feature detection unit 20 in which a robust feature detection method is implemented. Note that the feature detection unit 20 has modified the feature extraction units 11 and 12 in FIG. 1, the processing in 501 to 503 in FIG. 5, the processing in 601 to 603 in FIG. It is a configuration. The processing of the feature detection unit 20 can be realized by software program processing or hardware circuit processing. Moreover, the setting regarding the process of the feature detection part 20 is possible.

  In FIG. 8, the feature detection unit 20 is configured to include a region division unit 21, a feature extraction unit 22, and an unnecessary object removal unit 23. The area dividing unit 21 includes a road area detecting unit 21A and a roadside area detecting unit 21B. The road area detection unit 21A includes a road surface mark detection unit 21a and a lane detection unit 21b. The roadside area detection unit 21B includes a roadside band detection unit 21c. The unnecessary object removing unit 23 includes a road unnecessary object removing unit 23A and a roadside unnecessary material removing unit 23B. The unnecessary road removing unit 23A includes a dense horizontal line deleting unit 23a. The roadside unnecessary object removing unit 23B includes a dense corner point deleting unit 23b. Further, for example, a limiting unit 24 may be provided between the region dividing unit 21 and the feature extracting unit 22 (described later).

[Various areas and objects]
FIG. 9 shows examples of areas and objects on the road. This is a case of a two-lane road on one side. Broadly speaking, there are a road area 51 and a roadside area 52. 31 is a lane boundary line (white line etc.), 32 is a road surface traffic sign (for example, white line arrow). 41 is the own vehicle (running state), 42 is the other vehicle (running state, preceding vehicle), 43 is the other vehicle (running state, oncoming vehicle), and 44 is the other vehicle (parking and stopping state). Reference numeral 51 denotes a road area (including all lanes). Reference numeral 52 denotes a roadside area (non-road area) next to (side) the road area 51. Reference numeral 53 denotes a central separation band (or central boundary line). Reference numeral 54 denotes a travel lane area (for two lanes) including the host vehicle 41. Reference numeral 55 denotes an oncoming lane area (for two lanes). Reference numeral 56 denotes a travel lane (own lane) of the own vehicle 41. 57 is another lane next to 56. 58 and 59 are oncoming lanes (other lanes). Reference numeral 61 denotes a roadside belt or a sidewalk that can be next to or outside 57 (outer lane). Reference numeral 62 denotes a roadside non-road area (referred to as “roadside object area” for distinction from 52).

  Reference numeral 71 denotes a building existing on the roadside (62). The building 71 may or may not be a feature detection target as a landmark useful for positioning (here, it is a detection target). Reference numeral 72 denotes a tree existing on the roadside (62). 73 is a signboard (advertisement) present on the roadside (62). 74 is a person (such as a pedestrian) present in the roadside belt 61 or the like. 75 is a traffic sign, a traffic light, etc. which exist in the roadside (62).

  In this configuration, the “road area” detected by the road area detection unit 21A is, for example, 51, 54, 56 in FIG. The “roadside area” detected by the roadside area detection unit 21B is, for example, 52 (61, 62) in FIG.

  In this configuration, as objects to be detected / deleted as “unnecessary objects”, there are cars (42, etc.), trees 72, signboards 73, people 74, etc. (corresponding features). On the other hand, objects useful for positioning (corresponding features) include road white lines (31), traffic signs (32), road areas (51, etc.), landmark buildings 71, roadside traffic signs (75 )and so on.

[Example of actual image]
FIG. 10 shows a state before an area division / unnecessary object removal in an example of an actual image at a certain time / position, and FIG. 11 simply shows a state after the area division / unnecessary object removal in the actual image. This is the case of a one-lane road. As in FIG. 9, 31 is a white line (white line region), and 32 is a traffic sign. 81 is an own lane (travel lane of the own vehicle). 82 is an oncoming lane. Reference numeral 83 denotes a roadside belt next to the own lane 81. Reference numeral 84 denotes another vehicle (preceding vehicle) in front of the own lane 81. Although 72 is a tree, the outline and the like are simply shown.

  A broken line a indicates a line when the white line 31 is detected. A region defined by the white line 31 (a) is a lane (81). The area of A (A1) indicates the detected road area 51. The area of B (B1, B2) indicates the detected roadside area 52. The area C is an area other than A and B.

  In FIG. 9, the roadside zone 61 is defined to include the roadside band 61, and in FIG. 10, the roadside band 83 is defined to be included in the road area 51. These can have any definition.

[Feature detection processing]
The process of the feature detection unit 20 in FIG. 8 is as follows. The feature detection unit 20 inputs data of a real image 800 in which a road scene is captured.

  (1) The area dividing unit 21 performs a process of dividing the entire area of the frame of the input actual image (for example, FIG. 10) into various areas (for example, A, B, and C in FIG. 10). The area dividing unit 21 performs division into areas based on the road structure and the perspective transformation relationship of the camera 4. In addition, you may use (reference) the information of map DB5 in the case of area division.

  First, in the road area detection unit 21A, a road surface mark (useful information on traffic) including a white line (lane boundary line) 31 and a road surface traffic sign 32 is detected from the input real image area by the road surface mark detection unit 21a. The process is performed by a predetermined algorithm. For example, the road surface mark detection unit 21a detects an edge (a line segment or a corner point) such as a white line region (31) by using a Hough transform or the like by calculating a pixel difference value in the actual image data. Thereby, the white line 31 and the traffic sign 32 on a road surface are detected.

  The lane detection unit 21b detects a lane (road) including its own lane based on the detection result by the road surface mark detection unit 21a. In the example of FIG. 10, an area defined by the main white line 31 (a) can be detected as the own lane 81. Further, the area adjacent to the own lane 81 can also be detected as the oncoming lane 82 or the roadside band 83. When there are a plurality of lanes as shown in FIG. 9, a plurality of lanes can be detected by detecting each white line 31.

  Further, the road area detection unit 21A partitions a road area such as A according to the road structure and the perspective transformation relationship of the camera 4 (a perspective structure in which a three-dimensional space is viewed from the viewpoint of the camera 4 in the actual image). In the example of FIG. 10, the region near the far focus is divided so as to be included in the other region C.

  The road area detection unit 21A determines (divides) a road area 51 such as A (A1) in FIG. 10 based on the processing result. Then, the road area detection unit 21A outputs road area A information (referred to as r1).

  Further, the roadside region detection unit 21B uses the result (r1) of the road region detection unit 21A to estimate and detect the roadside region 52 including the roadside object region 62 from the input real image region by a predetermined algorithm. I do. For example, using the information on the shape of the road area A by r1 (including the shape of the white line 31 (a) etc.), along the shape of the road area A (or the white line 31 (a) etc.), for example, B in FIG. The roadside area 52 such as (B1, B2) is fixed (divided). Here, with respect to the Z direction (y direction) corresponding to the height from the road surface, the roadside area B is defined by a predetermined height restriction (for example, up to 4 meters) from the road surface. The height limit value can be set. Further, the roadside area detection unit 21B partitions a roadside area like B according to the road structure, the perspective transformation relationship, and the like. The processing efficiency can be improved by the height restriction.

  Further, the roadside zone detection unit 21c may be provided in the roadside region detection unit 21B (may be in the road region detection unit 21A). The roadside band detection unit 21c detects a roadside band area such as 61 in FIG. 9 and 83 in FIG. 10 by a predetermined algorithm (for example, estimation based on the outermost white line 31). It is determined to include it in the road area 51 or the roadside area 52.

  The roadside area detection unit 21B determines (divides) a roadside area 52 such as B (B1, B2) in FIG. 10 based on the processing result. Then, the roadside area detection unit 21B outputs roadside area B information (referred to as r2).

  In addition, the area C (FIG. 10) other than the areas A and B is an area where useful and important information for positioning is detected, such as traffic lights, traffic signs, corners of buildings that serve as landmarks, overpasses, and streetlights. Therefore, in the subsequent processing, the feature information (other than unnecessary items) is held as it is and used as an input in the positioning calculation. Other area C information (referred to as r3) is also output from the area dividing section 21 (roadside area detecting section 21B).

  (2) The feature extraction unit 22 inputs the actual image 800 data and the processing result information {road region A information (r1), roadside region B information (r2), other region C information (r3)} by the region dividing unit 21} To do. Similar to the above-described processing (feature extraction unit 11, measurement processing 503, 603, etc.), the feature extraction unit 22 is a feature group including corner points, line segments, color regions, and the like (feature amounts and position coordinates of each feature unit). (Including information) is performed. For example, a SIFT feature detector (resistant to rotation change and scale change), a Canny edge detector, a filter (color block region detection filter) for detecting a color block region by HSL color model conversion, and the like are used. The feature extraction unit 22 outputs the extracted feature part group information 801. Note that this information 801 includes information on a feature group extracted from each region (r1, r2, r3).

  In particular, the feature extraction unit 22 may perform effective feature extraction by changing a processing algorithm (filter or the like) applied to each region (A, B, C).

  (3) The unnecessary object removing unit 23 inputs the processing result information (801) of the feature extracting unit 22 and the information (r1, r2) of each region. Here, a case where the region C information (r3) is not input is shown. The feature part group extracted from the region C is retained as it is and is included in the output feature information 810.

  The road unnecessary object removing unit 23A receives the feature part group information 801 and the road area A information (r1), and performs a process of deleting unnecessary objects (corresponding characteristic parts) in the road area A. As the unnecessary items (unnecessary road items), for example, there are other vehicles 42 to 44 in FIG. 9, a person 74, another vehicle 84 in FIG. The dense horizontal line deleting unit 23a detects an area of a dense horizontal line (horizontal edge line segment) group from the road area A by a predetermined algorithm. The dense horizontal line deleting unit 23a detects a region having a large number of horizontal (x-direction) line segments as the feature group when the density is larger than a threshold value, for example. And since it can be estimated that the detected area is likely to be an area like the other car 84 in FIG. 10 (based on experiments and examinations by the present inventors), the unnecessary object removing unit 23 Are removed as unnecessary items (corresponding feature information is deleted).

  The roadside unnecessary object removing unit 23B receives the feature part group information 801 and the roadside area B information (r2), and performs a process of deleting unnecessary objects (corresponding characteristic parts) in the roadside area B. As the unnecessary items (roadside unnecessary items), for example, there are the other vehicle 44, person 74, tree 72, signboard 73, etc. in FIG. 9 (FIG. 10). The dense corner point deletion unit 23b detects a dense corner point (edge point) region from the roadside region B by a predetermined algorithm. The dense corner point deletion unit 23b detects a region having a large number of corner points as the feature unit group when the density is larger than, for example, a threshold value. And since this detected area | region can estimate that there is a high possibility that it is an area | region like the tree 72 (its outline) and the signboard 73 (its character string) of FIG. 10 (based on experiment and examination by the present inventors etc.). ), The unnecessary object removing unit 23 removes the area as an unnecessary object (deletes corresponding feature information).

  In the unnecessary object removing unit 23, as in the above-described process, the feature portion group corresponding to the unnecessary item is deleted from the input feature portion group information 801 to reduce the information / data amount, and the output feature portion group information 802 is output. And This information 802 becomes the feature information 810 output from the feature detection unit 20. In other words, in the present configuration (feature detection unit 20), the feature information (810) useful for the positioning calculation that does not depend on factors such as situation fluctuations is determined from the real image (800) and the feature group (801).・ Selecting is performed.

  FIG. 11 shows an example after unnecessary objects are removed from the real image of FIG. Features useful for positioning calculation, such as features corresponding to the white line 31 and traffic signs 32 in the region A, features corresponding to buildings 71 and traffic signs 75 in the roadside region B, and features in other regions C Information remains, and unnecessary objects (feature information that is not useful for positioning calculation) that vary greatly depending on the situation, such as other vehicles, trees, signboards, and the like in the areas A and B, are removed. 1101 etc. are examples of locations that have been deleted as unnecessary object areas.

  For example, the feature matching unit 13 performs positioning calculation using the feature information 810 output from the feature detection unit 20 as an input. As a result, it is possible to largely suppress (reduce) the adverse effect on the positioning calculation due to the removal of unnecessary objects, thereby realizing highly accurate positioning. Further, since the amount of information and data input for positioning calculation is reduced, the processing speed is also increased.

  In the above configuration example (unnecessary object removing unit 23), the application process for the area such that the deletion process of 23a is applied only to the road area A and the deletion process of 23b is applied only to the roadside area B. By limiting this to some extent, processing efficiency is improved. However, the present invention is not limited to this, and each deletion process can be applied to various areas.

[Limited part]
As a more applied form, a configuration provided with the limiting unit 24 of FIG. 8 will be described. The limiting unit 24 performs processing for limiting feature information or information (region or feature) used in the processing. Theoretically, positioning accuracy increases as the number of feature parts (feature points P) used for positioning calculation increases, but processing efficiency (processing time) tends to be worse (longer). Therefore, in this embodiment, based on the following method, the limiting unit 24 performs information limiting processing to reduce the information / data amount in the output feature information 810 (feature information used for positioning calculation), and processing Increase speed.

  (1) As a first method, the limiting unit 24 limits the number of feature parts (feature points P) included in an actual image (for each frame) to a certain number (maximum upper limit value). For example, the limiting unit 24 reduces the number of feature points P extracted from the real image area for each frame to a certain number (maximum upper limit value) in the process of extracting the feature group by the feature extraction unit 22. Restrict to. This fixed number is, for example, 50. This fixed number can be set. This information is deleted for features that exceed a certain number.

  (2) As a second method, the limiting unit 24 limits a target area in a subsequent process (including 22, 23, etc.) to a specific type area within the area of the input actual image 800. . In other words, the area other than the specific area is masked out of the processing target.

  In the real image of FIG. 12, 1200 indicates the horizon, and 1201 indicates the focus of the camera. 1202 is a region above the horizon 1200, and 1203 is a region below the horizon 1200. The limiting unit 24 limits the upper area 1202 as a process target area and masks the lower area 1203. As a result, the amount of information and data used in the feature extraction unit 22, the unnecessary object removal unit 23, and the positioning calculation is reduced, so that the processing becomes efficient.

  In addition to the above, the limiting unit 24 may change various ways of limiting the processing target area of the actual image. For example, it may be limited to [road area A + other area C] (excluding roadside area B). For example, the shape may be limited to a rectangle of a predetermined size around the focal point 1201 as in a range (area) 1204 in FIG.

[Effects]
As described above, according to the present embodiment, in the vehicle and in-vehicle information system 100, positioning of the current vehicle position (K) can be realized at low cost by a mechanism using the in-vehicle camera 4, and GPS or the like can be realized. It is possible to realize estimation (calculation) with higher accuracy than the above means. Compared to the conventional general positioning accuracy (10-15 meters), even the inexpensive camera 4 (rear view camera, etc.) and the existing inexpensive hardware / software for navigation are real time (output update: 10-60 Hz). ) Sub-meter positioning accuracy (average positioning error: 1 meter or less) can be achieved, and an advanced navigation interlocking system can be realized and can be spread to general vehicles.

  In particular, real-time positioning that does not use virtual image processing can be realized, and the pre-registration / update process can be facilitated / automated so that the production / maintenance of the map DB 5 can be made more efficient and less expensive. It is possible to provide a positioning technique including a technique such as robust feature detection that can be adapted to factors and fluctuations such as a road surrounding situation at the time of positioning (during imaging). Regarding the processing speed of real-time positioning, the average calculation time for one frame including image input can be shortened as compared with the conventional method.

  In particular, using the high-precision positioning result (L3) of the real-time positioning unit 1, for example, in the car navigation 6, as an element required for driving, at which intersection, which lane should be turned , Etc. can be navigated effectively. Further, it can be applied to a vehicle coordination system such as shift control to a vehicle, engine injection control, and inter-vehicle distance control in ACC.

  In the present invention, a two-dimensional real image (G) is used in the same manner as the conventional method 2 and the like, but high-precision absolute position information such as individual marks (feature points P and the like) is pre-registered in the map DB 5 and the like. There is no need to register (register position information estimated by the SFM method). Further, in the present invention, since objects (buildings, signboards, signs, etc.) existing on the roadside also incorporate many RP landscape features, the above-described unnecessary object removal processing causes the influence of shielding objects such as preceding vehicles (adverse effects on positioning accuracy). ) Can be minimized.

  In the present invention, high accuracy can be realized without requiring expensive means and devices as in the conventional method 3. The feature in the actual image is two-dimensional information, which has higher accuracy and lower ambiguity than the one-dimensional information by laser. Further, since the field of view of the camera 4 is large, it is difficult to be influenced by the preceding vehicle or other obstacles.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The present invention can be used for automobiles and in-vehicle information systems, car navigation systems, navigation-linked systems, vehicle advanced safety systems, and the like.

  DESCRIPTION OF SYMBOLS 1 ... Real time positioning part, 2 ... Control part, 3 ... Sensors, 4 ... Camera, 5 ... Map DB, 6 ... Car navigation, 10 ... Position detection part, 11 ... 1st feature extraction part, 12 ... 2nd Feature extraction unit, 13 ... Feature matching unit, 14 ... Position correction unit, 20 ... Feature detection unit, 21 ... Region division unit, 22 ... Feature extraction unit, 23 ... Unwanted object removal unit, 21A ... Road region detection unit, 21B ... Roadside area detection unit, 21a ... road surface mark detection unit, 21b ... lane detection unit, 21c ... roadside band detection unit, 23A ... roadside unnecessary material removal unit, 23B ... roadside unnecessary material removal unit, 23a ... dense horizontal line deletion unit, 23b ... Dense point deletion unit, 24 ... limitation unit, 100 ... automobile and in-vehicle information system.

Claims (9)

A moving body position measuring device,
A real-time positioning unit that measures or estimates the position of the moving object in real time;
The real-time positioning unit is
The first image is input from one or more imaging units that are mounted on the mobile body and capture at least one first direction around the mobile body in real time when the mobile body travels to obtain a first image. And
As the first image, a feature including a straight line or a corner point is extracted from a plurality of real images photographed in the vicinity of a reference point on the map, and used as first feature information. An extractor;
From the second image in the vicinity of a predetermined reference point on the map, a feature portion including a straight line or a corner point is extracted as second feature information, and data including the second feature information is registered in advance. A second feature extraction unit that reads out and obtains the corresponding second feature information in the vicinity of the reference point from the map database, when the mobile object is traveling;
By inputting the first feature information and the second feature information and comparing and matching the respective feature portions, the positions of the plurality of feature portions in the vicinity of the reference point are estimated, and the plurality of feature portions are estimated. Based on the position of the feature portion, the movement is calculated by calculating a relative positional relationship between the position of the moving body at each time point associated with the movement and the position of each of the plurality of feature portions using the SFM method. A feature matching unit that estimates the current position of the body and outputs it as second position information;
The first feature extraction unit or the second feature extraction unit includes:
A feature detection unit that inputs data of a real image captured on a map, detects a feature from the real image, and outputs feature information used in positioning calculation;
The feature detection unit includes a region division unit, a feature extraction unit, and an unnecessary object removal unit,
Said region dividing section, the image analysis processing, from the area of the real image, the road area, the roadside area and other area is detected by the division, the road area, the roadside area and other area Process to output the included information,
The feature extraction unit performs a process of extracting a feature group including straight lines and corner points from the real image region by image analysis processing and outputting feature information;
The unnecessary matter removing section, said at least one of the roadside area the road territory Ikima others in the real image, removed to detect a characteristic portion corresponding to the unnecessary substances for preventing use in the positioning calculation And a process of outputting the feature information after the removal. In the moving body position measuring apparatus of Claim 1,
Said region dividing section, the image analysis processing, from the area of the real image of the input, detecting a road area including the lane of travel of the vehicle, a road region processing for outputting information including the road area Having a detector,
The feature extraction unit, the image analysis processing, performed wherein the actual image road area or al, a process of outputting characteristic information by extracting a feature group including a linear or angular points,
The unnecessary matter removing section, the said in the real image road area or al, process the removal of the feature portions corresponding to the unwanted substances so that not used in the positioning calculation, and outputs the characteristic information after the removal A moving body position measuring apparatus comprising a road unnecessary object removing unit that performs the above-described operation. In the moving body position measuring apparatus of Claim 1,
It said region dividing section, the image analyzer processing, from the area of the real image of the input, before detecting a kilo side area, has a roadside area detection unit which performs a process for outputting information including the roadside area ,
The feature extraction unit, the image analysis processing, performed wherein the roadside area or we in the real image, the processing of outputting the feature information by extracting a feature group including a linear or angular points,
The unnecessary matter removing section, the said roadside area or we in the real image, processing the removed feature section that corresponds to the undesired substances in order not used in the positioning calculation, and outputs the characteristic information after the removal A moving body position measuring apparatus, comprising: a roadside unnecessary object removing unit that performs the above operation. In the moving body position measuring apparatus of Claim 2,
The road area detection unit
A road surface mark detection unit that performs a process of detecting a road surface mark including a white line from the region of the real image;
And a lane detector that performs a process of detecting one or more lanes based on the shape of the road surface mark from the area of the real image. In the moving body position measuring apparatus of Claim 3,
The roadside area detection unit, from said region of the real image, based on the shape of the road area, and based on the height limit from the road surface, by performing a process of detecting the roadside area, characterized by Mobile body position measuring device. In the moving body position measuring apparatus of Claim 2,
The road unnecessary matter removing section, the real image of the road area or al, detects an area in which a plurality of horizontal lines are gathered at a high density, by performing the process of deleting the feature groups corresponding to the detected area The moving body position measuring apparatus characterized by these. In the moving body position measuring apparatus of Claim 3,
The roadside unwanted matter removing section performs the roadside area or al of the actual image, detects a region where a plurality of corner points are gathered at a high density, the process of deleting the feature group corresponding to the detected region A moving body position measuring device. In the moving body position measuring apparatus of Claim 1,
The feature detection unit includes a limiting unit that performs a process of limiting the feature information for use in the positioning calculation,
The limiting unit performs a process of limiting the number of feature parts extracted from the real image region to a certain number in the process of the feature extraction unit. In the moving body position measuring apparatus of Claim 1,
The feature detection unit includes a limiting unit that performs a process of limiting the feature information for use in the positioning calculation,
The limiting unit performs a process of limiting a region above the horizon in the region of the real image as a processing target region, and deleting a region below the horizon as a processing target non-processing target. .