JP2011215974A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system suitable for creation of effective image data for reference since the creation of the effective image data for reference to be used for a scene image recognition technology becomes significant.SOLUTION: The image processing system recognizes, from a captured image of a scene viewed from a vehicle, a recognizable thing in the captured image as a feature object, and extracts one or more representative image feature points specifying the feature object, and generates feature point data including feature point position information indicating the position of the representative image feature point in the captured image and attribute information indicating object attributes as the attributes of the feature object. Furthermore, this system generates reference data for scene image matching by associating the image-capturing position with the feature point data.

Description

  The present invention relates to an image processing system, and more particularly to an image processing system that generates reference data suitable for vehicle position determination using image matching in a car navigation apparatus.

  Conventionally, in a car navigation apparatus, as a method of calculating the current position of a vehicle, a method using information acquired from a sensor such as a gyroscope or a geomagnetic sensor (autonomous navigation), a method using a signal from a GPS satellite, or an autonomous vehicle A method of combining navigation and GPS is adopted. Furthermore, in order to calculate the current position with high accuracy, the provisional current position is obtained using a signal from a GPS satellite, etc., and the provisional current position is used as a reference using the captured image in front of the vehicle. The coordinates of the feature points of the road markings (automotive coordinate system feature points) in the coordinate system (car coordinate system) are calculated, and the coordinates of the calculated feature points of the car coordinate system and the stored road marking feature points (world coordinates) There is known a position positioning device configured to calculate the current position of a vehicle using coordinates indicated by a system) (see, for example, Patent Document 1). In this apparatus, even if positioning based on signals from positioning satellites and signals from various sensors includes an error, it is possible to calculate a current position with high accuracy.

JP 2007-208043 (paragraph number 0009-0013, FIG. 1)

  However, in the positioning device according to Patent Document 1, the spatial coordinates of the feature points of the road marking on the road are obtained from the stereo image, and are obtained from the latitude / longitude of the road marking having the feature point stored in the road marking information database. Since the vehicle position is calculated using the coordinates obtained, it cannot be used in a place where there is no road marking. Further, since it is necessary to calculate the spatial coordinates of the feature points recognized by the image processing, the apparatus is required to have a high calculation capability, which causes a cost increase.

  Therefore, it is possible to use the landscape image recognition technique as a positioning that can be used on a road without a road sign or in a specific site, and does not have to calculate the spatial coordinates of each feature point for each position calculation. At that time, since it is important to create effective reference image data used in landscape image recognition technology, it is desired to realize an image processing system suitable for creating such effective reference image data.

  According to the present invention for solving the above-described problems, the characteristic configuration of an image processing system for creating reference data used when performing landscape image matching based on a captured image obtained by capturing a landscape from a vehicle is the captured image. And a data input unit for inputting a shooting position of the shot image, a feature object recognition unit for recognizing an identifiable thing in the shot image from the input shot image as a feature object, and the feature object from each of the feature objects A representative feature point extraction unit that extracts one or more representative image feature points that define a feature object, an attribute determination unit that classifies and determines an object attribute that is an attribute of each feature object, and Feature point position information indicating the position of the representative image feature point, and an object object to which the representative image feature point belongs. A feature point data generating unit that generates feature point data including attribute information that represents the object attribute; and a reference data generating unit that associates the shooting position with the feature point data and generates the reference data. .

  In the case of a non-simple composition such as a landscape image, the amount of processing becomes enormous in a matching process that compares pixel points between reference data and matching data as is normally done. However, according to the above configuration, by introducing the concept of a feature object that is identifiably recognized, one or more representative image feature points that define the feature object, and the attributes of the feature object are represented. Feature point data including attribute information can be used as reference data. The number of representative image feature points to be matched is a small number of representative image feature points, so that the calculation load is reduced. Of course, even if the matching target is a small number of representative image feature points, it is determined from the attribute information. In view of the feature object to be displayed, the representative image feature point is significant, so that sufficient matching accuracy can be ensured. As a result, by using the reference data generated in this way, it is possible to efficiently perform landscape image matching processing for specifying the shooting position, that is, specifying the vehicle position.

  Since a captured image with a landscape as a subject may include a plurality of feature objects of the same type, it is necessary to clearly distinguish these feature objects. For this reason, some information for distinguishing the same type of feature objects is necessary in the identification data defining the object attributes so that the recognized feature objects are not confused with other feature objects. For this purpose, it is preferable that the attribute information includes identification data assigned to each determined object attribute, and the identification data includes an identification code for distinguishing from other characteristic objects.

  A feature object that can be clearly defined with a small number of representative image feature points and is easy to process data is suitable. Therefore, in one preferred embodiment of the present invention, the feature object is a geometric shape, and the object attribute specifies a geometric shape. For example, a photographed image having an area such as a circle or a rectangle or a straight line intersecting with each other is relatively easy to recognize, and its definition formula is simple, so that it is easy to handle in terms of data processing. By replacing the landscape image with a geometric shape, the efficiency of the landscape image matching process is realized. At this time, when the recognized geometric shape is a specific geometric shape having a specific contour line, a geometric feature point located on the contour line is set as the representative image feature point, and the representative image feature point on the captured image is displayed. If the coordinate value is used as the feature point position information, the matching process is facilitated, which is convenient.

  There is a possibility that a feature object suitable for matching processing or a feature object unsuitable for matching processing may be recognized from a landscape image taken from a traveling vehicle. For example, a center line having a similar image pattern is not suitable for the matching process. Also, moving objects such as automobiles around the vehicle are not suitable for the matching process. Further, the position on the captured image where an important representative object is recognized differs depending on the area where the vehicle is traveling, for example, a mountain area, a suburb area, an urban area, a high-rise city area, or the like. Therefore, when shooting situation information that defines such a situation can be acquired, it is desirable to change the manner in which the feature object is extracted according to the shooting situation information. Accordingly, in one preferred embodiment of the present invention, shooting situation information indicating the possibility that a specific subject is included in the shot image is input, and the feature object is extracted from the shot image based on the shooting situation information. The extraction condition such as the extraction region from which is extracted is changed.

  In addition, it is preferable that the image feature point that is the basis of the representative feature point is one that can be stably detected at a point on the image. Therefore, in general, edge points that constitute an edge image obtained by using an edge detection filter or the like are used. Use it. Therefore, in a preferred embodiment of the present invention, the feature object is extracted from the captured image through an edge image obtained by performing edge detection processing on the captured image. ing.

  Furthermore, the present invention also includes a position positioning system that determines the position of the vehicle through matching processing using reference data created by the above-described landscape matching reference data generation system. Such a positioning system includes a reference data database storing the reference data, a data input unit for inputting an actual captured image obtained by capturing a landscape from a vehicle, and the captured image from the input actual captured image. A feature object recognition unit for recognizing an identifiable thing as a feature object, a representative feature point extraction unit for extracting one or more representative image feature points defining the feature object from each of the feature objects, An attribute determination unit that classifies and determines an object attribute that is an attribute of each feature object, feature point position information that represents the position of the representative image feature point in the actual captured image, and a feature object to which the representative image feature point belongs Generate feature point data including attribute information representing object attributes as matching data In addition to matching the reference data extracted from the reference data database and the matching data, the vehicle data position is determined based on the shooting position related to the reference data that has been successfully matched. And a landscape matching section. In this position positioning system, as described above, reference data effective for landscape image matching is used, so that the vehicle position can be determined satisfactorily.

  In addition, regarding the determination of the vehicle position, in one of the more preferable embodiments, the landscape matching unit selects the representative image feature point from each of a plurality of different feature objects included in the reference data. The feature point position information of each of the selected representative image feature points is read out, and the corresponding representative image feature points in the actual captured image are extracted, and the plurality of representative image features of the reference data are extracted. The vehicle position is determined based on the positional relationship between the points and the positional relationship between the corresponding representative image feature points extracted from the actual captured image. According to this configuration, based on the positional relationship between the plurality of representative image feature points in the reference data and the positional relationship between the corresponding representative image feature points extracted from the actual captured image, the reference data Since it is possible to calculate the difference between the shooting position and orientation of the original captured image and the shooting position and orientation of the actual captured image, it is possible to determine the position of the vehicle with higher accuracy.

It is a schematic diagram explaining the basic concept of the positioning technology which determines the own vehicle position through preparation of the reference data by the image processing system by this invention, and the matching process using the reference data. It is a functional block diagram which shows the main functions of the image processing system by this invention. It is a schematic diagram which shows the main processing functions and the flow of a process of a feature point extraction part. It is a schematic diagram which shows typically the process of producing | generating edge point data from a picked-up image. It is a schematic diagram which shows the process which produces | generates reference data from the edge point data in an image processing system. It is a functional block of a car navigation system using a reference data DB constructed by reference data created by an image processing system according to the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 schematically shows a basic concept of a position positioning technique for determining a vehicle position through landscape image matching.
A reference data database 92 (hereinafter simply referred to as “reference data DB”) used for landscape image matching can be constructed by creating a database of reference data generated by the image processing system according to the present invention. The construction procedure will be described below. As shown in FIG. 1, a photographed image obtained by photographing a landscape from a vehicle during traveling and a photographing position at the time of photographing (substantially the own vehicle position) are input and temporarily stored in a working memory (# 01). Next, an image feature is extracted from the photographed image developed in the working memory, and an identifiable thing in the photographed image is recognized as a feature object from such an image feature (# 02). These image features vary depending on the feature extraction method, such as those represented by image feature points, those represented by lines, and those represented by areas. The extraction result is represented by image feature points or feature line groups as a collection of image feature points. In this case, it is preferable that the feature object to be recognized is a geometric shape body defined by points or lines.

  Next, one or more representative image feature points that define the feature object are extracted from the recognized feature object (# 03). This representative feature point is an image feature point that defines a geometric shape if the feature object has a geometric shape. For example, in the case of a square, the four corner points are representative image feature points. When a representative image feature point is determined, an object attribute (attribute information) such as the type of geometric shape defined by the representative image feature point is classified and determined and combined with the coordinate position (feature point position information) of the representative image feature point. (# 04). Further, feature point data expressing all the feature objects composed of the feature point position information and the attribute information recognized in one photographed image is generated (# 05). The feature point data generated in this way is associated (linked) with the shooting position and becomes reference data for landscape image matching (# 06). Furthermore, the reference data is stored in the reference data DB 92 so as to be used as a pattern matching pattern in landscape image matching so that it can be searched and extracted using the shooting position as a search condition (# 07).

  Next, a procedure for determining the position of the vehicle (vehicle position) during actual vehicle travel using the reference data DB 92 constructed in the above-described procedure will be described. As shown in FIG. 1, an actual captured image obtained by capturing a landscape with an in-vehicle camera, and an imaging position and an imaging direction used for extracting reference data from the reference data DB 92 are input (# 11). ). The photographing position here is an estimated own vehicle position estimated using a GPS measurement unit or the like. From the input photographed image, matching data, which is feature point data, is generated through the processing steps of steps # 02 to # 05 (# 12). Further, using the input shooting position (estimated vehicle position) as a search condition, reference data of the corresponding shooting position and reference data before and after the shooting position (estimated vehicle position) are extracted as matching candidate reference data ( # 13).

  Reference data is set as a matching pattern one by one from the extracted matching candidate reference data sets, and pattern matching processing with the matching data generated at the present time is executed as landscape image recognition (# 14). If the matching is successful, the shooting position associated with the reference data that is the target is read (# 15), and this shooting position is determined as the official vehicle position that replaces the estimated vehicle position (#). 16).

Next, an example of an image processing system according to the present invention that creates reference data from a captured image based on the basic concept of the vehicle position measurement technique described above will be described with reference to the functional block diagram of FIG. This image processing system includes a data input unit 51, a feature point extraction unit 60, a feature object recognition unit 52, an attribute determination unit 53, a representative feature point extraction unit 54, an attribute information generation unit 55, a feature point data generation unit 56, and reference data. A generation unit 57 is provided. Each of these functions can be created in hardware or software or a combination thereof.
The data input unit 51 inputs a photographed image obtained by photographing a landscape from a vehicle traveling for the purpose of creating reference data and a photographing position of the photographed image. The feature point extraction unit 60 extracts an image feature suitable for landscape image recognition from a captured image, and sets a pixel or a pixel group indicating a predetermined image feature as an image feature point. In this embodiment, an edge point extracted based on an edge image obtained by performing edge detection processing on a captured image is an image feature point. Individually, the original photographed image is an RGB image, and the contour is detected by a luminance difference (density difference). Instead, a method in which the contour is detected by a saturation difference or a hue difference is used. It may be adopted. The extraction condition adjusting unit 60A limits the extraction, for example, area limitation, so that image specific points unsuitable for landscape matching do not remain in the end.

  The feature object recognition unit 52 has a function of recognizing an identifiable thing in the photographed image as a feature object from the image showing the image feature in the photographed image. Here, the feature object has a geometric shape. Yes. Here, as a geometric shape recognized as a feature object, basic geometric shapes such as two or more straight lines, circles, squares, and triangles having intersections are handled. The attribute determination unit 53 classifies and determines object attributes (line, circle, quadrangle, etc.) that are attributes of the feature object so that the geometric shape recognized by the feature object unit 52 can be used by other functional units. . Based on the classification determination result by the attribute determination unit 53, the representative feature point extraction unit 54 extracts one or more image feature points that define each feature object as representative image feature points. For example, in the case of a square, there are four corner points. The attribute information generation unit 55 generates attribute information by converting the object attribute including the classification determination result by the attribute determination unit 53 and the coordinate value of the representative feature point extracted by the representative feature point extraction unit 54 into attribute data. The feature point data generation unit 56 generates feature point data including feature point position information representing the position of the representative image feature point and feature object attribute information defined by the representative image feature point. The reference data generation unit 57 associates the shooting position (own vehicle position) with the feature point data generated by the feature point data generation unit 56 and generates reference data for landscape image matching.

Next, the data processing will be described in detail with reference to FIG. 3 schematically showing the main processing flow of the data input unit 51 and the feature point extraction unit 60.
The data input unit 51 includes a photographic image obtained by photographing a landscape by a camera mounted on a vehicle traveling for reference data creation, photographing attribute information including a photographing position and a photographing direction at the time of photographing, and a photographing situation. Information is entered. In the form in which the image processing system is mounted on the traveling vehicle, the captured image, the photographing attribute information, and the photographing situation information are input to the input unit 51 in real time. In such a configuration, the photographed image, photographing attribute information, and photographing state information are temporarily recorded on a recording medium, and these data inputs are performed in a batch process.

  The shooting situation information is information indicating the possibility that a specific subject is included in the shot image, and the contents included in the shooting situation information of this embodiment include travel lane data, moving object data, and area attribute data. It is. The traveling lane data is data indicating the traveling lane region of the own vehicle and the region outside the road in the photographed image obtained from the recognition result of the white line, the guide rail, and the safety zone obtained through image processing on the photographed image. The moving object data is data indicating an existing area in a captured image of a moving object present around the vehicle that is recognized by an in-vehicle sensor that detects an obstacle such as a radar. Area attribute data is data indicating area attributes such as mountain area, suburban area, urban area, high-rise city area, etc. is there.

  The feature point extraction unit 60 includes a temporary storage unit 61, an edge detection unit 62, an edge importance degree determination unit 63, a weighting unit 64, an adjustment coefficient setting unit 65, and an edge point image output unit 66. The temporary storage unit 61 is a working memory that temporarily stores an input captured image, and is normally assigned to a partial area of the main memory.

  The edge detector 62 extracts image feature points as image features from the captured image using an appropriate contour (edge) detection operator. The image feature points here are edge points and edge point groups (edge lines). The edge importance determination unit 63 determines the importance of the image feature points (edge points) extracted by the edge detection unit 62 based on the contents of each data included in the shooting situation information. For example, when the content of the travel lane data is used, higher importance is assigned to image feature points belonging to a region further away from the travel lane closer to the shoulder in the captured image. When moving object data is used, a low importance is assigned to image feature points belonging to a region where a moving object exists in a captured image. Further, when the contents of the area attribute data are used, the importance assigning rule corresponding to the position in the captured image is changed according to the area attribute. For example, in a captured image of a mountain area, since there is a high possibility that the upper part of the photographing center optical axis is empty and the left and right are forests, high importance is set for the central region around the photographing central optical axis. In the photographed image of the suburban area, there is little traffic of cars, and structures such as houses spread around, so a high importance is set for the area below the photographing center optical axis. In the captured image of the urban area, since there is a lot of traffic, a high importance is set for the area above the optical axis of the image capturing center. In the photographed image of the high-rise city area, since there are many elevated roads and viaducts, high importance is set for the region above the photographing center optical axis.

  The weighting unit 64 assigns weighting factors to the image feature points according to the importance determined by the edge importance determination unit 63. High importance is set for image feature points that are considered important for accurate image recognition (pattern matching), so a large weighting factor is assigned to image feature points with high importance. Considering that image feature points with low weighting factors are not likely to be used in actual image recognition or deleted from reference data, they can be used to determine the selection of image feature points Calculated.

  The adjustment coefficient setting unit 65 calculates an adjustment coefficient for changing the weighting coefficient assigned by the weighting unit 64 from the viewpoint of the distribution state in the corresponding captured image region. That is, the importance determined based on the shooting situation information for the image feature points extracted by the edge detection unit 62 includes a certain degree of error, and image feature points having a certain degree of importance are also randomly generated. There is also a possibility. For this reason, when there is an uneven distribution of image feature points, in other words, an uneven distribution of the weighting coefficient assigned by the weighting unit 64, the adjustment coefficient setting unit 65 is used to moderate the uneven distribution. When the distribution of image feature points obtained by calculation processing indicates the uneven distribution of image feature points, the adjustment coefficient is set so that the weighting factor of image feature points belonging to the region where the density of image feature points is small is increased. The adjustment coefficient is set so that the weighting coefficient of the image feature point belonging to the region where the density of the image feature points is large is small. The edge point image output unit 66 organizes the image feature points based on the weighting coefficient assigned by the weighting unit 64 and the adjustment coefficient that is optionally used, and outputs image feature point data for each captured image. That is, the edge importance determination unit 63, the weighting unit 64, and the adjustment coefficient setting unit 65 function as an extraction condition adjustment unit 60A that adjusts the extracted image feature points.

  Here, the process of spreading image feature points as widely as possible over the entire captured image area using the above-described adjustment coefficient will be described with reference to the schematic explanatory view shown in FIG. An edge detection image (FIG. 4B) is generated by extracting an image feature point (edge point) from the photographed image (FIG. 4A). Importance is given to each edge point of the edge detection image. 4C shows the importance corresponding to each edge point in the form of an importance layer corresponding to the feature point image so that the appearance of the importance can be schematically understood. In this importance layer, a weighting factor is assigned to each edge point, as shown in Fig. 4D, in the form of an edge point image in which the edge point is drawn so that the larger the weighting factor, the larger the point becomes. Here, when the edge points are arranged such that the edge points to which the weighting coefficient equal to or less than the predetermined threshold is assigned are removed, for example, FIG. ) Only edge points that are large points are selected. In this case, the edge points located in the lower area of the edge point image are excluded, and a large uneven distribution occurs in the distribution of the remaining edge points. An adjustment coefficient that increases the weighting coefficient of the edge point in the area where the density of the edge point that is calculated and selected as a result is low is set so that the adjustment coefficient that is set can be understood schematically. In FIG. 4E, the adjustment coefficient group is shown in the form of an adjustment coefficient layer arranged in a matrix (in this case, in units of sections made up of a plurality of pixel areas) so as to correspond to the edge point image. The point image output unit 66 organizes each image feature point using the weighting factor finally set based on the weighting factor and the adjustment factor, and the edge as shown in FIG. Shooting point images It is generated for each.

  Here, the degree of importance is determined for each image feature point (edge point), and as a result, a weighting factor is set for each image feature point. However, these processes may be performed in units of groups. Is possible. In that case, for example, the captured image area is divided into a plurality of image sections, and the edge importance determination unit 63 groups the image feature points belonging to the same image section as an image feature point group and handles them in a unified manner. The image feature points included in the image feature point group are given the same importance, and the weighting unit 64 may similarly set the weighting coefficient in units of image feature point groups. In addition, the image sections handled here may be handled in units of one pixel constituting the captured image, but may be handled in units of a plurality of pixels.

Processing for recognizing a geometric shape as a feature object from the edge point image output from the above-described feature point extraction unit 60, specifying an edge point included in the edge point image by this geometric shape, and generating feature point data The flow will be described with reference to the schematic diagram of FIG.
The feature object unit 52 recognizes a geometric shape defined by one or more edge points in the edge point image obtained from the photographed image, for example, intersecting straight lines, triangles, squares, circles, and the like. Such a geometric shape can be recognized, for example, by recognizing a straight line or a circle by Hough transform of the image. By recognizing a plurality of straight lines and their intersections, triangles (three intersections) and quadrangles (four intersections) can also be recognized. These recognition algorithms are well known and will not be described in detail. When a specific geometric shape is recognized, the attribute determination unit 53 determines an object attribute that is information for specifying the geometric shape, and the representative feature point extraction unit 54 defines one specific geometric shape. Alternatively, two or more edge points are specified and extracted as representative image feature points, and their coordinate values are captured. Then, the attribute information generating unit 55 creates attribute information for each geometric shape having various parameters specifying the recognized geometric shape (feature object) as attribute values. For example, if the recognized geometric shape is a quadrangle, the representative image feature points are four corner points (edge points) of the quadrangle. In the case of a circle, the representative image feature point is an edge point located at the intersection of the X axis parallel line passing through the center of the circle and the Y axis parallel line and the circumference. If two lines intersect, the intersection may be used as a representative image feature point, and the intersection angle may be attached as a descriptive feature. Here, a simple line segment is also treated as a recognized geometric shape, and the representative image feature points are an edge point on one end and an edge point on the other end of the line segment. As is apparent from the above description, generally, when the recognized geometric shape is a specific geometric shape having a specific contour line, an edge point (geometric feature) located on the contour line that is the edge point group. Point) is adopted as the representative image feature point, and the coordinate value of the representative image feature point on the captured image is incorporated as feature point position information in the form of an attribute value.
Further, the attribute information given for each recognized geometric shape includes an identification code, that is, an ID code, in order to distinguish it from other geometric shapes of the same type.
Further, if a recognition process using a plurality of images is adopted as a geometric shape recognized on the image, a light emitting area shape created by a light-emitting body that periodically blinks can be included. In such an amorphous shape, it is convenient to use the center of gravity as an attribute value.

  The feature point data generation unit 56 includes position information (for example, an edge point image) that represents the coordinate position of an edge point that represents the position of the representative image feature point, and the geometric shape as a feature object defined by the representative image feature point. Feature point data including attribute information is generated. The reference data generation unit 57 associates the shooting position (own vehicle position) with the feature point data generated by the feature point data generation unit 56 and generates reference data for landscape image matching.

  In addition, in the above-described reference data generation process, in order to limit (extraction conditions) extraction of feature objects from a captured image based on shooting state information indicating the possibility that a specific subject is included in the captured image, In advance, the restriction based on the shooting state information is performed in advance when the edge point image is generated. That is, the edge points of such areas are excluded in advance so that the feature objects are not recognized from the areas that are deemed inappropriate for matching, as determined from the shooting situation information. Instead of this method, when recognizing a feature object, an inappropriate region may be specified from the shooting situation information and excluded from the feature object recognition target.

  Next, an in-vehicle car navigation system that corrects the vehicle position by landscape image recognition (image feature point pattern matching) using the reference data DB 92 created by the above-described image processing system will be described. FIG. 6 is a functional block showing such a car navigation system incorporated in an in-vehicle LAN. This car navigation system includes an input operation module 21, a navigation control module 3, a vehicle position detection module 4, a photographing situation information generation unit 7, a road map database 91 (hereinafter referred to as a road map database 91) that stores the reference data DB 92 and car navigation road map data. And a database 9 having simply a road map DB).

  The navigation control module 3 includes a route setting unit 31, a route search unit 32, and a route guide unit 33. The route setting unit 31 sets, for example, a departure point such as the vehicle position, an input destination, a passing point, and traveling conditions (whether or not an expressway is used). The route search unit 32 is a processing unit that performs arithmetic processing for searching for a guide route from the departure point to the destination based on the conditions set by the route setting unit 31. The route guidance unit 33 provides appropriate route guidance to the driver by guidance display on the display screen of the monitor 12 or voice guidance by the speaker 13 according to the route from the departure place to the destination searched by the route search unit 32. It is a process part which performs the arithmetic processing for performing.

  The own vehicle position detection module 4 corrects the estimated own vehicle position obtained by the conventional position calculation by GPS and the position calculation by dead reckoning with the own vehicle position determined by landscape image recognition using the estimated own vehicle position. It has the function to do. The own vehicle position detection module 4 includes a GPS processing unit 41, dead reckoning processing unit 42, own vehicle position coordinate calculation unit 43, map matching unit 44, own vehicle position determination unit 45, captured image processing unit 5, and landscape matching unit 6. I have. A GPS measurement unit 15 that receives GPS signals from GPS satellites is connected to the GPS processing unit 41. The GPS processing unit 41 analyzes the signal from the GPS satellite received by the GPS measurement unit 15, calculates the current position (latitude and longitude) of the vehicle, and sends it to the own vehicle position coordinate calculation unit 43 as GPS position coordinate data. The dead reckoning processing unit 42 is connected to the distance sensor 16 and the azimuth sensor 17. The distance sensor 16 is a sensor that detects a vehicle speed or a moving distance of the vehicle. For example, a vehicle speed pulse sensor that outputs a pulse signal every time a drive shaft, a wheel, or the like of the vehicle rotates by a certain amount detects the acceleration of the host vehicle C. And a circuit that integrates the detected acceleration. The distance sensor 16 outputs information on the vehicle speed and moving distance as the detection result to the dead reckoning processing unit 42. The direction sensor 17 includes, for example, a gyro sensor, a geomagnetic sensor, an optical rotation sensor attached to the rotating part of the handle, a rotary resistance volume, an angle sensor attached to the wheel part, and the like, and the direction of the detection result Information is output to dead reckoning processing unit 42. The dead reckoning processing unit 42 calculates dead reckoning position coordinates based on the moving distance information and the direction information sent every moment, and sends the dead reckoning position coordinate data to the own vehicle position coordinate calculating unit 43 as dead reckoning position coordinate data. The own vehicle position coordinate calculation unit 43 performs an operation for specifying the position of the vehicle from the GPS position coordinate data and the dead reckoning position coordinate data by a known method. The calculated vehicle position information is information including measurement errors and the like, and in some cases, the vehicle position information is off the road. Therefore, the map matching unit 44 determines the vehicle position on the road indicated on the road map. Correction is performed. The vehicle position coordinates are sent to the vehicle position determination unit 45 as the estimated vehicle position.

  The photographed image processing unit 5 substantially includes most of the functional units that constitute the image processing system shown in FIG. The captured image processing unit 5 includes a data input unit 51, an edge detection unit 62, an edge importance determination unit 63, a weighting unit 64, an adjustment coefficient setting unit 65, and an edge point image output unit 66. When the photographed front landscape photographed image from the vehicle is input to the data input unit 51, the image feature point data is output from the edge point image output unit 66 through the procedure described above. Note that the shooting situation information used in the edge importance determination unit 63 is generated by the shooting situation information generation unit 7 mounted on the vehicle and is sent to the shot image processing unit 5. The shooting state information generation unit 7 is connected to the in-vehicle camera 14 to generate the travel lane data, and receives the same shot image sent to the shot image processing unit 5. Traveling lane data is created by image processing the received captured image using a known algorithm. The photographing situation information generation unit 7 is connected to an obstacle detection sensor group 18 in order to create the moving object data. Moving object data is created based on sensor information from the sensor group 18. Further, the shooting situation information generation unit 7 is connected to the vehicle position determination unit 45 and the database 9 in order to create the above. The database 9 is searched using the vehicle position coordinates and the search key from the vehicle position determination unit 45 to acquire area attributes (mountainous area, city area, etc.) of the current traveling location, and the area attribute data is based on the area attributes. Created.

  The scenery matching unit 6 uses the reference data extracted from the reference data DB 92 based on the estimated vehicle position sent from the vehicle position determination unit 45 as a pattern, and the image feature points sent from the captured image processing unit 5 Perform pattern matching processing on data. If the pattern matching is successful, the shooting position associated with the reference data that is the matching pattern is read out. This photographing position is transferred to the own vehicle position determination unit 45 as the own vehicle position. The own vehicle position determination unit 45 corrects the own vehicle position by replacing the transferred own vehicle position with the estimated own vehicle position.

  This car navigation system also includes an input device 11 such as a touch panel 11 or a switch as a peripheral device, and an operation input evaluation unit 21a that changes an operation input through the input device 11 into an appropriate operation signal and transfers the operation signal to the inside. An input operation module 21, a display module 22 for displaying image information necessary for car navigation on the monitor 12, a voice generation module 23 for sending voice information necessary for car navigation from the speaker 13 or a buzzer, braking, acceleration, steering, etc. The vehicle behavior detection module 24 that detects various behaviors of the vehicle based on behavior data sent through the in-vehicle LAN is provided.

(Another embodiment)
Although not employed in the above-described embodiment, as another embodiment of the present invention, each of the plurality of feature objects included in the reference data regarding the determination of the vehicle position by the vehicle position detection module 4 It is also preferable that the detailed vehicle position is determined based on the positional relationship between the representative image feature points and the positional relationship between the corresponding representative image feature points extracted from the actual captured image. In this case, the landscape matching unit 6 selects a representative image feature point from each of a plurality of different feature objects included in the reference data, and reads out feature point position information of each of the selected representative image feature points. In addition, the landscape matching unit 6 extracts a plurality of corresponding representative image feature points in the actual captured image. Then, the landscape matching unit 6 determines the vehicle position based on the positional relationship between the plurality of representative image feature points in the reference data and the positional relationship between the corresponding representative image feature points extracted from the actual captured image. To do. Specifically, based on the difference between the positional relationship between the plurality of representative image feature points of the reference data and the positional relationship between the corresponding plurality of representative image feature points extracted from the actual captured image, the reference data is generated. The deviation of the captured position and orientation of the actual captured image from the captured position and orientation of the captured image is calculated. Thereby, the landscape matching unit 6 determines the own vehicle position with higher accuracy.

In the above-described embodiment, the image feature point is an edge point obtained by edge detection processing, particularly a line segment edge constituting one line segment or an intersection point where such a line segment intersects, preferably approximately orthogonally. A corner edge is treated as an effective image feature point. However, the present invention is not limited to such edge points as image feature points. For example, typical edge points that form geometric shapes such as circles and quadrilaterals (three points on the circumference of a circle if they are circles), the center of gravity of an amorphous geometric shape, and the point as the center of gravity Is used because it is an effective image feature point. It is also preferable to adopt edge strength as a factor for calculating importance. For example, if a line segment is a strong edge, the start point and end point of the line segment are treated as highly important image feature points. be able to. In addition, specific points in a characteristic geometric shape, such as edge points and barycentric points of symmetrical objects, can be handled as highly important image feature points.
Furthermore, in addition to the edge points obtained by the edge detection process, it is also possible to regard a captured image as a change in hue or saturation and to employ a point with a large change as an image feature point. Similarly, an edge point of an object having a high color temperature can be handled as a highly important image feature point based on color information.
That is, the image feature points handled in the present invention are all used if they are effective for similarity determination between the reference data and the image feature amount data generated from the actual captured image, for example, pattern matching. It becomes.

In the above-described embodiment, the reference data stored in the reference data DB 92 is associated with the shooting position and the shooting direction (camera optical axis direction). The date and time and the weather at the time of shooting may also be related.
The photographing position may be two-dimensional data such as latitude / longitude data at least, but may be three-dimensional data in addition to height data.
Further, it is not essential to relate the shooting direction to the reference data. For example, when creating reference data and when recognizing a landscape image using this reference data, the shooting direction is not required if it is guaranteed that the road is shot with substantially the same shooting direction. It becomes.
Conversely, in the case where reference data in which the shooting direction is appropriately shifted from the reference data in one basic shooting direction can be prepared, based on the traveling direction of the vehicle calculated from information such as a direction sensor, It is also possible to set only the reference data suitable for the traveling direction as the object of landscape image recognition.
The on-vehicle camera handled in the present invention is most suitable for photographing a landscape in front of the vehicle traveling direction. However, the camera may be a camera that takes a landscape oblique to the front, or may be a camera that captures a landscape of the side or rear. That is, the photographed image handled in the present invention is not limited to only a photograph of a front landscape in the vehicle traveling direction.

  The functional units shown in the functional block diagrams used in the description of the above-described embodiments are for easy understanding, and the present invention is not limited to the divisions shown here. It is possible to freely combine function parts or further divide one function part

  The image processing system of the present invention can be applied not only to car navigation but also to a technical field that measures the current position and orientation by landscape image recognition.

3: navigation control module 4: own vehicle position detection module 41: GPS processing unit 42: dead reckoning processing unit 43: own vehicle position coordinate calculation unit 44: map matching unit 45: own vehicle position determination unit 5: image processing system 51: Data input unit 52: feature object recognition unit 53: attribute determination unit 54: representative feature point extraction unit 55: attribute information generation unit 56: feature point data generation unit 57: reference data generation unit 60: feature point extraction unit 61: temporary storage Unit 62: Edge detection unit 63: Edge importance determination unit 64: Weighting unit 65: Adjustment coefficient setting unit 66: Edge point image output unit 6: Landscape matching unit 14: Camera 92: Reference data DB (reference data database)
91: Road map DB (Road map database)

Claims (8)

An image processing system for creating reference data used when performing landscape image matching based on a photographed image of a landscape from a vehicle,
A data input unit for inputting the photographed image and the photographing position of the photographed image;
A feature object recognition unit for recognizing an identifiable thing in the captured image as a feature object from the input captured image;
A representative feature point extraction unit that extracts one or more representative image feature points defining the feature object from each of the feature objects;
An attribute determination unit that classifies and determines an object attribute that is an attribute of each of the characteristic objects;
A feature point data generation unit that generates feature point data including feature point position information representing the position of the representative image feature point in the captured image and attribute information representing the object attribute of the feature object to which the representative image feature point belongs When,
A reference data generating unit that associates the shooting position with the feature point data and generates the reference data;
An image processing system.   The image processing system according to claim 1, wherein the attribute information includes identification data assigned for each determined object attribute, and the identification data includes an identification code for distinguishing from other characteristic objects.   The image processing system according to claim 1, wherein the feature object is a geometric shape, and the object attribute specifies a geometric shape. When the recognized geometric shape is a specific geometric shape having a specific contour line, a geometric feature point located on the contour line is set as the representative image feature point, and a coordinate value on the captured image of the representative image feature point The image processing system according to claim 3, wherein the feature point position information is used.   The shooting condition information indicating the possibility that a specific subject is included in the shot image is input, and an extraction condition for extracting the feature object from the shot image is changed based on the shooting condition information. 5. The image processing system according to any one of items 1 to 4.   The image processing system according to claim 1, wherein the extraction of the feature object from the photographed image is performed through an edge image obtained by performing an edge detection process on the photographed image. In the position positioning system which determines the own vehicle position through the matching process using the reference data created by the image processing system according to any one of claims 1 to 6,
A reference data database storing the reference data;
A data input unit for inputting an actual captured image of a landscape from the vehicle;
A feature object recognition unit for recognizing an identifiable thing in the photographed image as a feature object from the input photographed image;
A representative feature point extraction unit that extracts one or more representative image feature points defining the feature object from each of the feature objects;
An attribute determination unit that classifies and determines an object attribute that is an attribute of each of the characteristic objects;
Feature point data including feature point position information representing the position of the representative image feature point in the actual captured image and attribute information representing the object attribute of the feature object to which the representative image feature point belongs is generated as matching data. A feature point data generation unit;
A landscape matching unit that performs matching between the reference data extracted from the reference data database and the matching data, and determines a vehicle position based on a shooting position related to the reference data that has been successfully matched. Positioning system.   The landscape matching unit selects the representative image feature point from each of a plurality of different feature objects included in the reference data, reads the feature point position information of each of the selected plurality of representative image feature points, A plurality of corresponding representative image feature points in the actual captured image are extracted, the positional relationship between the plurality of representative image feature points in the reference data, and the corresponding plurality of representative image features extracted from the actual captured image. The position positioning system according to claim 7, wherein the position of the vehicle is determined based on the positional relationship between the points.

Images