JP2012203723A - Feature information learning system, feature information learning program, and feature information learning method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively reflect the condition of sunshine at the time of recognizing an image to develop, for recognizing the image, feature information including information indicating the effect of sunshine when recognizing the image.SOLUTION: For each of features, feature shade information being information indicating a relationship between a feature and a shade is calculated; contrast information indicating the contrast between a sunny place and a shade is acquired; sunshine cause information including the feature shade information and the contrast information and an image recognition result of an image recognition part performing image recognition of a feature included in a photographed image are associated with each other to be stored in a learning-database; and for each of the features, sunshine characteristic information indicating a degree of influence of sunshine to image recognition is generated on the basis of the information stored in the learning-database to store the generated information as information constituting feature information on each of the features.

Description

  The present invention relates to a feature information learning system, a feature information learning program, and a feature information learning method for learning feature information that is information relating to a feature to be image-recognized.

  Navigation systems that guide the route of people and vehicles are widely used. In such a navigation system, the position information obtained by surveying using GPS (global positioning system) and the relative position information obtained by using dead-reckoning technology are used to determine the course. invite. In addition, there is a case where position information is calculated by image processing to suppress errors included in position information obtained by surveying using GPS or dead reckoning, and to further improve the accuracy of position information. For example, the position information is calculated by recognizing features such as paint (road markings), signboards, signs, and the like on the road whose existence position is known in advance from the captured image. In Japanese Patent Laid-Open No. 2008-299650 (Patent Document 1), highly accurate self-location information is acquired by recognizing such a feature, and route guidance or the like is obtained based on the high-precision self-location information. A navigation system that performs navigation processing is exemplified (67th paragraph, etc.).

  However, since image recognition generally requires a relatively high calculation load, it is not realistic to recognize all the features that are assumed to be present in the captured image. Therefore, in Patent Document 1, by appropriately selecting (extracting) features suitable for use in image recognition processing, erroneous recognition is suppressed and image recognition is performed efficiently (fifth paragraph). , Summary, etc.). For example, features that are extremely faint and are not suitable for image recognition, or features that have a low recognition rate indicating the success probability of image recognition are excluded, and features that are most easily recognized are selected (paragraphs 102 to 204, etc.). . However, in image recognition, the influence on the recognition result by sunlight, for example, such as the difference in brightness of images to be recognized, is not small. Therefore, in consideration of the sunshine situation at the time of image recognition, a feature different from the above feature may be the most suitable for image recognition. In view of this point, when selecting a target feature to be subjected to image recognition from a plurality of features whose presence is known in advance, the situation of solar radiation at the time of performing image recognition processing is also effectively reflected. Thus, it is preferable to select a more optimal target feature. And in selecting the target features, in order to effectively reflect the state of solar radiation, it is necessary to maintain feature information including information indicating the influence of solar radiation when recognizing the feature in each feature. desirable.

JP 2008-299650 A

  In view of the above background, feature information including information indicating the influence of solar radiation at the time of image recognition is prepared in order to effectively reflect the sunshine situation at the time of image recognition and to perform image recognition. Technology is desired.

A feature information learning system that learns feature information that is information about a feature that is subject to image recognition,
A captured image acquisition unit for acquiring captured images;
An image recognition unit for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation unit for calculating feature / shade information, which is information indicating a positional relationship between the feature and the shade,
A contrast information acquisition unit that acquires contrast information indicating contrast between the shade and the sun;
A learning storage unit that stores sunshine factor information including the feature shade information and the contrast information and an image recognition result of the feature by the image recognition unit in association with each other in a learning database;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning unit that stores information as constituent information;
It is in the point provided with.

  Image recognition of features is often performed by performing known image processing such as edge extraction and pattern matching on a captured image. For example, edge extraction is calculated according to a difference or gradient of pixel values such as luminance and density indicated by adjacent pixels or pixel groups. For this reason, in addition to the pixel value difference between the feature to be image-recognized and the background of the feature, if there is a difference in pixel value between the sun and the shade, edge extraction may become inaccurate. is there. In addition, if the edge detection becomes inaccurate, there is a possibility that the success rate of the pattern matching performed after that will also decrease. In particular, the success rate of image recognition differs depending on the positional relationship between the feature (candidate feature) that is a candidate for the target feature and the shade. Also, the success rate of image recognition varies depending on the difference in pixel values between the sun and shade, that is, the contrast. The learning storage unit has the sunshine factor information including the feature shade information (information indicating the positional relationship between the candidate feature and the shade) and the contrast information, which has a great influence on the image recognition, and the image recognition unit. The image recognition result is associated and stored in the learning database. The feature information learning unit generates sunshine characteristic information indicating the degree of influence on image recognition based on the information stored in the learning database, and stores it as information constituting the feature information. Therefore, the feature information including information that affects the success rate of the image recognition as described above is well prepared.

  By the way, the feature information can be collected, for example, by running a vehicle dedicated to collecting information, but in cooperation with an image recognition system that actually recognizes the feature using the feature information. It is also possible to collect them. In general, when a feature is image-recognized and the accuracy of position information is improved, the target feature to be image-recognized is often selected from a plurality of candidate features. That is, when image recognition is performed on the target feature, information such as the location and type of the other feature is acquired, and a captured image including the other feature is also acquired. Since it is acquired, it is efficient when collecting feature information. Further, if the image recognition of the target feature is successful, the position information becomes more accurate information, and therefore it is possible to accurately grasp the existence position of the feature other than the target feature, that is, the learning feature. As a result, it is possible to maintain feature information efficiently with high accuracy.

As such one aspect, the feature information learning system according to the present invention is:
A photographing device for photographing the photographed image;
An estimated position information acquisition unit that acquires estimated position information indicating an estimated position of the imaging device;
A temporary storage unit that accumulates the captured image over a predetermined period and temporarily stores the image in association with the estimated position information;
A candidate feature that is connected to a feature database that stores the feature information including at least the position information of the feature and that is the feature information of a plurality of candidate features that exist along the movement path of the imaging device A feature information acquisition unit for acquiring information,
The image recognition unit performs image recognition of one target feature selected from the plurality of candidate features, and when image recognition of the target feature is successful, The estimated position information stored in the temporary storage unit is corrected based on the position information included in the feature information, and the captured image is read from the temporary storage unit based on the corrected estimated position information. , Based on the candidate feature information, image recognition of the learning feature that is the candidate feature different from the target feature included in the captured image,
Preferably, the learning storage unit is configured to store the result of the image recognition and the sunshine factor information in the learning database in association with each other.

  By the way, in order to store the sunshine factor information including the feature shade information and the contrast information and the image recognition result by the image recognition unit in association with each other in the learning database, for each feature, the feature and shade It is necessary to satisfactorily calculate the feature shade information that is information indicating the relationship. And the image recognition part needs to perform image recognition of each learning feature in consideration of this feature shade information. Therefore, as one aspect, the feature information learning system according to the present invention is a sunshine state detection unit that detects the sunshine state at each position in association with the route position information indicating the position along the movement route of the imaging device. The feature shade information calculation unit includes the feature for each of the features based on the detection result of the sunshine state detection unit and the position information of the feature included in the feature information. Shade information is calculated, and the image recognition unit is stored in the temporary storage unit based on the position information included in the feature information about the learned feature and the corrected estimated position information. It is preferable to perform image recognition of the learning feature included in the captured image.

  Various technical features of the feature information learning system according to the present invention described above can be applied to a feature information learning program and a feature information learning method. Therefore, the present invention can also target such a feature information learning program and a feature information learning method. For example, the feature information learning program according to the present invention can cause a computer to realize various functions having the features of the feature information learning system described above. The typical aspect is illustrated below. As a matter of course, such a feature information learning program can also exhibit the effects of the above-described feature information learning system. Furthermore, various additional features exemplified as a preferred mode of the feature information learning system can be incorporated into the feature information learning program, and the program also exhibits the effects corresponding to the respective additional features. Can do. The same applies to the feature information learning method according to the present invention.

As one aspect, the feature information learning program according to the present invention includes:
A feature information learning program for learning feature information, which is information about a feature to be image-recognized,
A captured image acquisition function for acquiring captured images;
An image recognition function for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation function for calculating feature / shade information, which is information indicating the positional relationship between the feature and the shade,
A contrast information acquisition function for acquiring contrast information indicating contrast between the shade and the sun;
A learning storage function for associating and storing in the learning database the sunshine factor information including the feature shade information and the contrast information, and the image recognition result of the feature realized by the image recognition function;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning function to store as constituent information;
Is preferably realized by a computer.

Moreover, as one aspect, the feature information learning method according to the present invention includes:
A feature information learning method for learning feature information, which is information about a feature to be image-recognized,
A captured image acquisition step of acquiring a captured image;
An image recognition step for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation step for calculating feature / shade information, which is information indicating the positional relationship between the feature and the shade,
A contrast information acquisition step of acquiring contrast information indicating contrast between the shade and the sun;
A learning storage step of associating and storing a sunshine factor information including the feature shade information and the contrast information in a learning database in association with an image recognition result of the feature by the image recognition step;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning step to store as constituent information;
It is suitable that it is provided.

Block diagram schematically showing the system configuration of the navigation system Explanatory drawing showing an example of the layer structure of the navigation database Flow chart schematically showing an example of the entire procedure for selecting features The flowchart which shows an example of the procedure which acquires sunshine factor information typically A flowchart schematically showing an example of a procedure for calculating the degree of influence on image recognition A flowchart schematically showing an example of a procedure for selecting a target feature Explanatory drawing which shows an example of the feature which exists in a predetermined range Explanatory diagram showing an example of shadows on features Correlation diagram showing an example of feature information of candidate features Figure showing an example of selecting features according to the sunshine factor The figure which shows the example which selects the feature with respect to the sunlight factor different from FIG. The figure which shows the example which selects a feature with a different method according to the same sunshine factor as FIG. Flow chart schematically showing an example of a procedure for learning feature information The flowchart which shows an example of the procedure which produces | generates sunshine characteristic information typically

  Hereinafter, an embodiment of the present invention will be described with an example of a feature recognition system for recognizing features such as paint on a road surface (road marking) in order to improve the accuracy of vehicle position information in a vehicle navigation system. explain. In the following description, road markings will be described as examples of features. However, installations where the relative position to the ground is fixed, including signs such as road signs that are installed on the roadside or suspended across the road. Objects can also be features. In the following, a vehicle navigation system equipped with such a feature recognition system will be described as an example. However, this navigation system is not limited to a so-called in-vehicle navigation system that is fixedly mounted on a vehicle, but can be arbitrarily selected. It may be a portable navigation system that can be moved. Further, it may be a navigation system installed as an application in a portable device such as a mobile phone or a digital camera.

[Navigation system]
FIG. 1 is a block diagram schematically showing the system configuration of such a navigation system. As shown in FIG. 1, the navigation system 1 includes a self-position specifying system 2 and a guide system 7. The guide system 7 is a system that is the core of the guidance function of the navigation system 1, and provides route guidance to the destination and various information provisions at the own position based on the position (own position) where the navigation system 1 exists To do. The self-localization system 2 is a system that identifies the position (local position) where the navigation system 1 exists, and is self-localization using surveying by GPS (global positioning system) or autonomous control by dead-reckoning. And a positioning system 3 that calculates the position of the vehicle by recognizing the above-described features and calculating its own position.

  The “own position” is a position that is defined in each system, particularly the navigation system 1, can be recognized by a user who receives guidance by the navigation system 1, and can appropriately use the guidance result. For example, when the navigation system 1 is an in-vehicle navigation system fixedly mounted on a vehicle and the user is a passenger of the vehicle, the “own position” is the position of the navigation system 1 and the vehicle It is a position, a position of an imaging device such as an in-vehicle camera, and a user's position. When the navigation system 1 is a portable navigation system, the “own position” is at least the position of the navigation system 1 and the position of the user who carries the portable navigation system. When a camera (photographing device) is attached to the portable navigation system, the position of the camera is also “own position”. When the portable navigation system is brought into the vehicle, the position of the vehicle is naturally included in the “own position”. Furthermore, when the portable navigation system can acquire a photographed image by a photographing device such as a vehicle-mounted camera mounted on the vehicle, the position of the vehicle-mounted camera is also included in the own position. The “local position” of a navigation system installed as an application in a portable device such as a mobile phone or a digital camera is the same as that of the portable navigation system, and thus detailed description thereof is omitted.

  The feature recognition system 4 includes a feature selection system 5 and an image processing system 6. The feature selection system 5 is a system that selects a target feature that is a target of image recognition by the image processing system 6 from a plurality of features whose presence positions are known in advance. The image processing system 6 is a system that recognizes an image of the target feature selected by the feature selection system 5. In the present embodiment, the own position is first estimated by the estimated position calculation unit 31 of the positioning system 3. Here, this own position is referred to as “estimated position information”. Based on the recognition result of the target feature by the feature recognition system 4, the estimated position information is corrected to “high-accuracy position information” indicating the self-position with high accuracy in the position information correction unit 32 of the positioning system 3. The guide system 7 performs route guidance to the destination and provision of various information at the own position by using at least the estimated position information, and more preferably using the high-accuracy position information.

  In the present embodiment, the navigation system 1 includes an arithmetic processing device such as a microcomputer or a DSP (digital signal processor) as a core member. That is, the navigation system 1 is configured by cooperation of hardware such as an arithmetic processing unit, a storage medium such as a memory or a disk device, a peripheral IC, and software such as programs and parameters used on the hardware. The The same applies to each of the above-described systems 2 to 7 constituting the navigation system 1 and the feature information learning system 8 described later.

  As shown in FIG. 1, the navigation system 1 includes a navigation database (navigation DB) 11. For example, the navigation database 11 is realized by an optical disk device, a magnetic disk device, an SSD (solid state drive), or the like. In the present embodiment, the navigation database 11 is a database in which a plurality of map information divided for each predetermined area and a plurality of feature information associated with the map information are stored. The navigation database 11 is constructed with a layer structure as shown in FIG. In the present embodiment, an example in which the road network layer m1, the road shape layer m2, and the feature layer m3 are stored in the navigation database 11 is shown.

  The road network layer m1 is a layer indicating connection information between roads. Specifically, it has information on a large number of nodes n having position information on a map expressed by latitude and longitude, and information on a large number of links k that form a road by connecting two nodes n. It is configured. Each link k has information such as road type (type of highway, toll road, national road, prefectural road, etc.) and link length as link information. The road shape layer m2 is a layer stored in association with the road network layer m1 and indicating the shape of the road. Specifically, information on a number of road shape complementing points s, information on road widths, and the like, which are arranged between two nodes n (on link k) and have position information on a map expressed by latitude and longitude, It is configured. Map information is constituted by information stored in the road network layer m1 and the road shape layer m2.

  The feature layer m3 is configured in association with the road network layer m1 and the road shape layer m2, and is a layer in which information of various features provided on the road and around the road, that is, the feature information F is stored. is there. The feature in which the feature information F is stored in the feature layer m3 includes a road marking provided on the road surface. Examples of such features related to road markings include pedestrian crossings, stop lines, traffic direction markings that specify the traveling direction of each lane (straight arrows, right-turn arrows, etc.), intersection shape markings (cross marks, T mark, etc.), lane markings dividing the lane along the road (solid line, broken line, double line, etc.), various display marks such as speed display, zebra zone and the like are included. In addition to such road markings, the features in which the feature information F is stored can include various features such as traffic lights, signs, overpasses, and tunnels. As described above, the detailed feature information F is stored in the navigation database 11. It can also be said that the navigation database 11 includes at least a map database 11m and a feature database 11f as shown in FIG. In the present embodiment, the navigation database 11 also includes a three-dimensional object database 11b described later, and stores three-dimensional object information. Although illustration is omitted, in addition to the layers m1 to m3 described above, a three-dimensional object layer in which three-dimensional object information is stored may be provided.

  The feature information F is information including position information about at least a position where the feature exists. This position information includes information on the position (latitude and longitude) of the representative point of each feature on the map and the direction of each feature. The representative point of the feature is set, for example, at the center position in the length direction and the width direction of each feature. The feature information F includes attribute information of each feature. In the present embodiment, the attribute information includes information on a feature type, a feature form, a feature state, and recognition characteristics. The feature type information is information representing the type of each feature, and basically features of the same shape are defined as one type. For example, the feature type information is information indicating a specific type of road marking such as a straight arrow, a right turn arrow, a cross mark, a T-shaped mark, a stop line, and a pedestrian crossing. The feature form information includes information such as the shape, size, and color of each feature. The feature state information includes feature state information that affects the image recognition processing of each feature, such as the actual fading state of each feature. In addition, the present embodiment includes sunshine characteristic information and reference recognition rate information as recognition characteristic information. The sunshine characteristic information is information on characteristics indicating the degree of influence of each day on image recognition by sunshine. The reference recognition rate information is information indicating an average success probability of image recognition for each feature.

  As described above, the navigation system 1 includes the own position specifying system 2 and the guide system 7, and the guide system 7 uses the information on the own position specified by the own position specifying system 2, Carry out route guidance. The self-localization system 2 includes a positioning system 3 and a feature recognition system 4. The feature recognition system 4 includes a feature selection system 5 and an image processing system 6. ing. By recognizing the target feature selected by the feature selection system 5 by the image processing system 6, the positioning system 3 corrects the error included in the positioning result and obtains high-accuracy position information.

  In the present embodiment, the positioning system 3 includes an estimated position calculation unit 31 and a position information correction unit 32. First, the estimated position information is calculated and acquired by the estimated position calculation unit 31. . As shown in FIG. 1, the estimated position calculation unit 31 is connected to the GPS receiver 12, the azimuth sensor 13, and the distance sensor 14. The estimated position calculation unit 31 can also acquire information such as the traveling direction of the vehicle and the moving speed by accumulating information on the current position. The position information correction unit 32 corrects the estimated position information using information on the position with higher accuracy, and provides the corrected position information to the guide system 7. When information regarding such a highly accurate position is not obtained, for example, when a feature is not recognized by the image processing system 6, the estimated position information is provided to the guide system 7 as it is.

  Note that high-precision position information is always calculated and does not need to be used for the guide system 7. For example, a configuration in which high-accuracy position information is calculated only when high-accuracy position information is requested by an upper control function unit (including software) of the navigation system 1 may be employed. In general, since image processing is computationally expensive, it may not be practical to construct a system that can always perform computations to obtain highly accurate position information. Therefore, when the accumulated error assumed to be included in the position information by the estimated position calculation unit 31 satisfies a certain condition such as exceeding a predetermined threshold, the feature recognition system 4 is used to obtain a high-precision position. It is preferred if the information is computed. In addition, when highly accurate guidance such as a course change point is necessary, or when highly accurate position information is required to assist braking on a stop line, regardless of the accumulated error included in the position information, The high-precision position information may be calculated using the feature recognition system 4 preferentially.

  By the way, since image processing, particularly image recognition, is generally a processing with a large calculation load, recognizing all the features whose existence is known is from the viewpoint of calculation time, performance and scale of the calculation device. Not realistic. For this reason, the feature recognition system 4 includes a feature selection system 5 for selecting a target feature to be subjected to image recognition from candidate features that are candidates for the target feature. In the present embodiment, the feature selection system 5 uses the estimated position information calculated by the estimated position calculation unit 31 of the positioning system 3 to select the target feature to be recognized. As shown in FIG. 1, the feature selection system 5 includes a date / time information acquisition unit 41, an estimated position information acquisition unit 42, a feature information acquisition unit 43, a three-dimensional object information acquisition unit 44, and a contrast information acquisition unit 51. And a selection calculation unit 50 including a halation information acquisition unit 52, a feature shade information calculation unit 53, and a target feature selection unit 58. The feature selection system 5 includes a picked-up image acquisition unit 61 shared with the image processing system 6 in addition to the selection calculation unit 50. Hereinafter, an example of a procedure for selecting a target feature will be described using the flowcharts of FIGS. Detailed description of each of the above-described functional units and other functional units will be appropriately performed in the following description of the procedure for selecting the target feature.

[Selection of target features (feature selection system)]
When high-precision position information is requested by a higher-level control function unit (including software) of the navigation system 1, the estimated position information is acquired by the estimated position information acquisition unit 42 (FIG. 3 # 01: estimated position information acquisition) Step / function). The “local position” indicated by the estimated position information can be the “positions of various objects” as described above, but is the “local position” when recognizing a feature using a captured image. Then, it is information indicating the estimated position of the imaging device that captures at least a captured image used for image recognition. In the present embodiment, the photographing apparatus is the camera 16 shown in FIG. The camera 16 uses a CCD (charge coupled device), CIS (CMOS image sensor), or other imaging device to shoot a two-dimensional image of 15 to 30 frames per second in time series, and digitally converts it into moving image data (captured image). ) In real time. The camera 16 is, for example, a wide-angle camera having a viewing angle of 140 to 190 ° in the horizontal direction, and is installed in a vehicle having a depression angle of about 30 degrees on its optical axis, and is about 8 m from the vehicle. An area can be photographed. When the camera 16 is installed in a rear bumper or a rear door of the vehicle as a rear camera that captures the rear of the vehicle, the camera 16 outputs an image including a road surface behind the vehicle as a captured image. In addition, the front camera which image | photographs the front of a vehicle may be provided as an imaging device, and the picked-up image of the said front camera may be used. Such a front camera is often installed with the optical axis facing upward as compared with the above-described rear camera, and a feature different from a road marking such as a road sign can be well included in the field of view. Needless to say, captured images of both the front camera and the rear camera may be used.

  The feature information acquisition unit 43 sets a predetermined range (search range) that is a range for searching for candidate features that are candidates for the target feature that is the target of image recognition based on the estimated position information (search range setting). Step / function). FIG. 7 shows an example of the feature f in the search range SP. Here, an example is shown in which six features of the features f1 to f6 are included in the search range SP as features whose presence is known in advance. Subsequently, the feature information acquisition unit 43, from the feature database 11f (navigation database 11), candidate feature information that is the feature information F for a plurality of candidate features existing within the search range SP (predetermined range). (FIG. 3 # 02: feature information acquisition step / function). In the case of this embodiment, the search range SP is set based on the estimated position information, and the feature f existing in the search range SP is set as the candidate feature, and the feature information F of the candidate feature (candidate (Feature information). Therefore, the feature information acquisition step (function) performed by the feature information acquisition unit 43 includes the search range setting step (function) and the search range SP (in the traveling direction of the camera 16 based on the estimated position information). It can also be said that the candidate feature information is acquired from the feature database 11f using the feature f existing within the predetermined range) as the candidate feature.

  When the feature information F of the candidate feature is acquired, the target feature selection unit 58 that selects the target feature from the candidate features determines whether or not the target feature needs to be selected (# 03 in FIG. 3). : Selection necessity judgment step / function). If there is no candidate feature within the search range SP, selection is not necessary, and the selection process is terminated. Strictly speaking, even when there is one candidate feature, it is not necessary to select one candidate feature, and it is sufficient to use the one candidate feature as the target feature. However, in the present embodiment, the target feature is determined after evaluation of suitability for image recognition based on at least a general condition. Therefore, even when there is one candidate feature, it is determined that the target feature needs to be selected.

  If it is determined in step # 03 that the target feature needs to be selected, the target feature selecting unit 58 narrows down the candidate features based on the general conditions. In other words, the target feature selection unit 58 selects a candidate feature from the features f whose existence is grasped in the search range SP (FIG. 3 # 04: Candidate feature selection step / function). The feature information F includes information on the feature f, such as the state of the feature, for example, the degree of fading of the road marking. For a feature f with a large amount of blur, there is a high possibility of image recognition failure or misrecognition as another feature even if the feature f is a target feature. There is a case. Therefore, the target feature selecting unit 58 selects candidate features in the search range SP based on general conditions such as a blurred state. Moreover, the feature f3 and the feature f4 which show the rhombus-shaped pedestrian crossing notice shown in FIG. 7 are two continuous features. Such a feature f is preferably combined to form a feature set g1 (g) in order to improve recognition accuracy. The target feature selecting unit 58 may reorganize the features in the search range SP into the feature set g based on general conditions such as combination conditions and select candidate features.

  Subsequently, the target feature selection unit 58 determines whether or not there are a plurality of selected candidate features (# 05 in FIG. 3). If there are not a plurality of candidate features, it is next determined whether there is one candidate feature or zero (# 06). When the candidate feature is zero, there is no feature f that is the target feature, and the processing of the entire feature selection system 5 is ended. Since there is no room for selection when there is one candidate feature, the target feature selection unit 58 selects the candidate feature as the target feature (# 08). On the other hand, when there are a plurality of candidate features, the target feature selecting unit 58 stops the determination for selecting the target feature, and calculates and acquires various information necessary for the selection. 5, particularly the other functional units of the selection calculation unit 50. Specifically, various information such as a photographed image, date / time information, and sunshine factor information is acquired, and a target feature is selected in consideration of the influence of sunshine on image processing. Hereinafter, acquisition of these various information and selection of the target feature in consideration of the influence of sunlight will be described. It should be noted that the order of acquiring various information shown in the following Step # 07, Step # 09, and Step # 20 is an example set for convenience of explanation, and there is no problem even if it is other orders.

  First, as shown in FIG. 1, a captured image of the camera 16 is acquired by a captured image acquisition unit 61 that is also a functional unit of the image processing system 6 (FIG. 3 # 07: captured image acquisition step / function). As described above, since the camera 16 outputs a two-dimensional image of 15 to 30 frames per second taken in time series as moving image data, it is preferable that a predetermined number of frames are temporarily stored in a frame memory (not shown) or the like. It is. At this time, if the frame is appropriately thinned and stored according to the unit movement amount of the camera 16 that moves with the vehicle using the detection results of the direction sensor 13 and the distance sensor 14, the capacity of the frame memory is suppressed. be able to. In addition, you may utilize the temporary memory part 81 of the feature learning system 8 mentioned later as this frame memory. Next, date / time information indicating the current date / time is acquired by the date / time information acquisition unit 41 (# 09 in FIG. 3: date / time information acquisition step / function). This date / time information is preferably acquired based on the GPS signal received by the GPS receiver 12, as shown in FIG.

  Subsequently, sunshine factor information is acquired by various functional units constituting the selection calculation unit 50 (FIG. 3 # 20: sunshine factor information acquisition step / function). The sunshine factor information is a factor affecting the image recognition of the feature f by the image recognition unit 63 of the image processing system 6 and is information indicating a state caused by sunshine. The sunshine factor information includes at least feature shade information and contrast information. The sunshine factor information preferably includes halation information. The feature shade information, which will be described in detail later, is information indicating the relationship between the candidate feature and the shade. Preferably, at least the feature is in the shade state where all the features are included in the shade, and all the features are included in the sun. This is information that can distinguish the sunshade state and the partial shade state where a part of the feature is included in the shade. Similarly, although details will be described later, the contrast information is information indicating the contrast in the captured image, that is, the shade and the sunny contrast that appear in the captured image or are estimated to appear in the captured image. Further, the halation information is information indicating the state of occurrence of halation in the captured image. Hereinafter, although the procedure of acquiring sunshine factor information will be described using the flowchart of FIG. 4 as well, the order of step # 22 to step # 24 is not particularly limited in this case.

  In order to accurately acquire the feature shade information that is one of the sunshine factor information and the contrast information, information on the position of the shadow estimated to occur in the captured image is necessary. And it is solid objects, such as a building around a feature, that becomes a big factor which forms such a shadow, ie, the shade contained in a photography picture. Therefore, the three-dimensional object information acquisition unit 44 acquires three-dimensional object information about a plurality of three-dimensional objects existing within the search range SP (predetermined range) from the three-dimensional object database 11b (navigation database 11) (# 21 in FIG. 4). Solid object information acquisition step / function). The three-dimensional object information includes information on the position of the three-dimensional object, information on the size (width, depth, height), information on the shape (such as a rectangular parallelepiped shape, a tower shape, a rod shape, and an inclined roof). Since the position of the sun is specified from the date and time information, the function unit constituting the feature selection system 5 is based on the date and time information and the three-dimensional object information, and the shade formed in the search range SP as shown in FIG. The position and shape of SH can be calculated. The relationship between the date / time information and the position of the sun is preferably stored as a map in the navigation database 11, the memory constituting the feature selection system 5, or the like. Needless to say, shaded SH information corresponding to the date / time information may be stored in the three-dimensional object database 11b.

  For example, based on the position and shape of the shade SH, the contrast information acquisition unit 51 calculates and acquires the contrast between the shade and the sun from a captured image when the vehicle 100 travels along the alternate long and short dash line Y shown in FIG. be able to. Further, when the vehicle 100 travels, based on the sunshine state information detected by the sunshine state detection unit 55 based on the illuminance detected by the illuminance sensor 17 and the position and shape of the shade SH, the shade and the sun The contrast can be calculated and acquired. In this way, the contrast information acquisition unit 51 acquires the contrast information indicating the contrast between the shade and the sun included in the captured image (FIG. 4 # 22: contrast information acquisition step / function).

  The halation information acquisition unit 52 calculates and acquires the state of occurrence of halation in the photographed image by determining whether or not the luminance information is saturated in the photographed image (# 23 in FIG. 4: obtaining halation information). Step / function). Since halation due to solar radiation basically occurs when it is exposed to sunlight, it is preferable that the halation information acquisition unit 52 calculates the occurrence state of halation at a position corresponding to the sunlight in the captured image.

  The feature shade information calculation unit 53 is information indicating the relationship between the candidate feature and the shade SH for each of the candidate features based on the candidate feature information that is the feature information related to the candidate feature and the date and time information. The feature shade information is calculated and acquired (FIG. 4 # 24: feature shade information calculation step / function). As described above with reference to FIG. 8, the position and shape of the shade SH formed in the search range SP are clarified based on the date / time information indicating the position of the sun and the three-dimensional object information. The positional relationship with the object is also clarified. Referring to FIG. 8, an arrow feature f6 is partially shaded with the shade SH1 of the three-dimensional object 201, and the feature shade information is “partial shade state” (hereinafter referred to as “partial shade” as appropriate). Abbreviated). Since the faint arrow feature f5 is excluded from the candidate features, the feature shade information is not calculated. The diamond-shaped feature f4 of the pedestrian crossing notice is covered with the shade SH1 of the three-dimensional object 201, and the feature shade information is “shaded state” (hereinafter, abbreviated as “shade” as appropriate). Similarly, the shade 202 of the three-dimensional object 202 is hung on the diamond-shaped feature f3, and the feature shade information is “shaded state”. In the example shown in FIG. 8, since the feature set g1 is also included in the shade SH1 and the shade SH2, the feature shade information of the feature set g1 is also in the “shade state”. If the three-dimensional object 202 does not exist, the shade SH1 is applied to a part of the feature set g1, so the feature shade information of the feature set g1 is “partial shade”. The feature f2 of the stop line and the feature f1 of the pedestrian crossing are not covered with the shade SH3 of the three-dimensional object 203 present in the vicinity, and the shade information of these features is in the “sunshine state” (hereinafter, “ Abbreviated as "Hyuga").

  In this way, when the sunshine factor information is acquired, the target feature selection unit 58 selects the target feature from the candidate features based on the sunshine factor information including at least the feature shade information and the contrast information. Select (target feature selection step / function). The target feature selection step / function includes step # 30 and step # 60 shown in FIG. In step # 30, the degree of influence on image recognition is calculated (influence degree calculating step / function). In step # 30, the degree of influence at the time of image recognition of each candidate feature is calculated for each candidate feature based on the sunshine factor information of each target feature. In step # 60, one target feature is selected from the candidate features based on the degree of influence (feature determination step / function). That is, after the calculation of the degree of influence on all candidate features is completed, step # 60 (feature determination step / function) is performed, and one target feature is selected from the candidate features.

  First, step # 30 (influence degree calculating step / function) will be described with reference to the flowchart of FIG. The feature image recognition is performed by performing known image processing such as edge extraction and pattern matching on the photographed image. For example, the edge extraction is calculated according to a difference in pixel values such as luminance and density, a gradient, or the like indicated by adjacent pixels or pixel groups. For this reason, in addition to the difference in pixel value between the object (here, the feature) and the background (here, the road surface), if there is a difference in the pixel value between the sun and the shade, the edge extraction becomes inaccurate, and the subsequent pattern The success rate of matching etc. may also decrease. In particular, if the difference in pixel values between the sun and the shade, that is, the contrast is large, there is a high possibility that edge extraction will be inaccurate. Accordingly, referring to the sunshine factor information (feature shade information) of the candidate feature that is the calculation target of the influence degree, it is determined whether or not the attribute of the candidate feature is the feature f corresponding to “partial shade”. First, determination is made (FIG. 5 # 31). If the attribute of the candidate feature is partial shade, the magnitude of the contrast is determined with reference to the sunshine factor information (contrast information) (# 34). If the contrast is large, the edge extraction is likely to be inaccurate as described above, so it is determined that the influence on the image recognition is large (# 41). When the contrast is small, the influence on the pixel value due to the shade is small, so it is determined that the influence on the image recognition is small (# 42).

  On the other hand, if it is determined in step # 31 that it is not a partial shade, it is next determined by referring to the feature shade information whether the attribute of the candidate feature is “shade” (# 32). ). If the attribute of the candidate feature is “shade”, the sunshine characteristic information that is included in the feature information and indicates the attribute specific to the feature is referred to, and “whether the shade has an effect on image recognition” is Determination is made (# 33). The sunshine characteristic information is information indicating the degree of the influence on the image recognition by the sunshine including the degree of the influence on the image recognition when the feature shade information is in the shade state for each of the features f. For example, if the feature is in the shade and the contrast between the feature and the background decreases, it affects the recognition rate of the feature, or if the feature is in the sun and affected by halation, the feature is recognized. The degree of influence on the rate is defined as sunshine characteristic information. In the present embodiment, when the feature is in the shade, the influence on the recognition rate due to the lack of contrast is exemplified as the sunshine characteristic information simplified to “with influence” and “without influence”. Accordingly, in step # 33 of the present embodiment, it is determined whether or not there is an influence due to insufficient contrast. This sunshine characteristic information is acquired from the feature database 11 f by the feature information acquisition unit 43.

  FIG. 9 shows an example of candidate feature information that is the feature information about the candidate feature acquired from the feature database 11f. Here, as the feature information, type information, reference recognition rate information, and sunshine characteristics (influence on image recognition when the feature shade information is in a shaded state) are shown. The reference recognition rate information is information indicating an average success probability of image recognition for each feature. 7 and 9, the type of the feature f1 is a pedestrian crossing road marking, the standard recognition rate is 70%, and the sunshine characteristic is “insufficient contrast”. The type of the feature f2 is a road marking of a stop line, the standard recognition rate is 60%, and the sunshine characteristic is “insufficient contrast”. The types of the feature f3 and the feature f4 are road markings for pedestrian crossing notice, the standard recognition rate is 90%, and the sunshine characteristics are “insufficient contrast”. The type of the feature f6 is an arrow road marking, the reference recognition rate is 50%, and the sunshine characteristic is “none”.

  By the way, for example, when the difference in pixel value between the object (here, the feature) and the background (here, the road surface), that is, the contrast due to the shade, the edge extraction may become inaccurate. As described above, the sunshine characteristic information indicating the influence on the image recognition when the feature shade information is in the shaded state includes information such as “contrast decreases due to shade”. Furthermore, when the entire captured image is bright, that is, in a clear state and bright except in the shade, the dynamic range of the entire captured image is generally adjusted by the camera 16. As a result, the number of shades in the shaded portion of the photographed image is relatively reduced, which may further reduce the contrast. On the other hand, when the entire captured image is dark, that is, when it is cloudy and dark except for the shade, the adjustment of this dynamic range does not reduce the number of shades in the shaded portion of the captured image. There is a possibility that the difference in pixel value between the background and the background, that is, the contrast does not decrease.

  If it is determined in step # 33 that “the shade does not affect the image recognition”, the candidate feature is determined to be “no influence on the image recognition based on the sunshine factor information” (# 43). . On the other hand, if it is determined in step # 33 that "the image recognition is affected by the shade", in step # 34 described above, the contrast between the shade and the sun is referred to with reference to the sunshine factor information (contrast information). Large or small is judged. When the contrast is large, there is a high possibility that the entire captured image is bright. As described above, the number of shades in the shaded portion of the photographed image is reduced, and there is a possibility that the difference in pixel value between the object and the background, that is, the contrast is lowered. Therefore, if it is determined in step # 34 that the contrast between the shade and the sun is large, it is determined that the influence on the image recognition is large (# 41). If it is determined in step # 34 that the contrast between the shade and the sun is small, it is determined that the influence on the image recognition is small because only the decrease in contrast due to the shade affects the image recognition (#). 42).

  If it is determined in step # 32 that the attribute of the candidate feature is not “shade”, in this embodiment, the attribute of the candidate feature is “Hyuga”. In the case of Hinata, halation may occur in the captured image due to the relationship between the sun position (irradiation angle) relative to the camera 16 and the reflection coefficient such as the road surface. When the position where halation occurs overlaps with the position of the feature, the pixel value of the photographed image causes saturation, the contrast between the feature and the road surface also decreases, and the success rate of image recognition decreases. For this reason, if it is determined in step # 32 that the attribute of the candidate feature is not “shade”, then the halation occurs at the position of the candidate feature with reference to the halation information (sunshine factor information). Is determined (# 35). If no halation has occurred, it is determined that “the degree of influence = none” since there is no influence on image recognition (# 43). On the other hand, if halation has occurred, the size of the halation is determined (# 36). When the halation is large, it is determined that “the degree of influence = high” (# 44), and when the halation is not large, it is determined that the “degree of influence = small” (# 45).

  As described above, the target feature selection unit 58 proceeds with selection of target features by lowering the priority of candidate features whose feature shade information indicates a partial shade state. In addition, the target feature selection unit 58 advances the selection of the target feature by lowering the priority of candidate features whose feature shade information indicates a partial shade state as the contrast between the shade and the sun indicated by the contrast information is larger. To go.

  The determination results of step # 41 to step # 45 described above, that is, the calculation result of the degree of influence are stored for each candidate feature (# 47). Then, it is determined whether or not the calculation of the influence level for all candidate features has been completed (# 48). If there is a candidate feature for which the calculation of the influence level has not been completed, step # 31 described above is performed. The calculation in step # 47 is performed on the incomplete candidate feature. When the calculation of the influence degree for all candidate features has been completed, the candidate features are classified into a plurality of groups according to the influence degree of each candidate feature (# 49). In the present embodiment, candidate features with “influence degree = none” are classified into group A, candidate features with “influence degree = low” are classified into group B, and candidate features with “influence degree = high” are classified into group C. Of course, this classification is an example, and it does not prevent the classification into four or more groups or the classification into two groups. Further, it goes without saying that the determinations exemplified in step # 31 to step # 45 can be appropriately changed according to the number of classifications.

  After the calculation of the degree of influence on all candidate features is completed, step # 60 (feature determination step / function) is performed, and one target feature is selected from the candidate features. Details of step # 60 will be described below using the flowchart of FIG. First, it is determined whether or not there is a group A composed of candidate features of “degree of influence = none” (# 61). If group A exists, group A is selected as the group to which the candidate feature selected as the target feature belongs (# 63). If the group A does not exist, the group B composed of candidate features of “influence degree = small” is selected (# 64). If the group B does not exist, the group C composed of candidate features of “influence degree = high” is selected (# 65). Then, the candidate feature having the highest reference recognition rate is selected as the target feature from the candidate features belonging to the group selected as the group to which the candidate feature selected as the target feature belongs (# 66). That is, if there is one candidate feature belonging to the selected group, the candidate feature is selected, and if there are a plurality of candidate features belonging to the selected group, a plurality of candidates The candidate feature with the highest reference recognition rate among the features is selected.

  Here, the selection of the target feature will be described with a specific example. FIG. 10 shows an example of selecting features according to the sunshine factor. Referring to FIGS. 5, 8, and 10, the feature f1 that is the road marking of the pedestrian crossing and the feature f2 that is the road marking of the stop line have the feature shade information “Hyuga” and the halation information “None”. Since the feature shade information is “Hyuga”, it is not necessary to consider the sunshine characteristic information and the contrast information, and the degree of influence on the image recognition is “None”. Therefore, the feature f1 and the feature f2 are classified into the group A. For the feature f3 and the feature f4, which are road markings for pedestrian crossing notice, the feature shade information is “shade” and the contrast information is “large”. Further, the sunshine characteristic information of the feature f3 and the feature f4 is “insufficient contrast”. Since the feature shade information is “shade”, there is no need to consider halation information, and the degree of influence on image recognition is “large” based on the feature shade information and the sunshine characteristic information. Therefore, the feature f3 and the feature f4 are classified into the group C. The feature shade information of the feature f6 that is the road marking of the arrow is “partial shade”, and the contrast information is “large”. Since the feature shade information is “partial shade”, it is not necessary to consider the sunshine characteristic information and the halation information, and the feature f6 is classified into the group C based on the feature shade information and the contrast information.

  In the description so far, as illustrated in FIG. 5, an example is shown in which each information is sequentially determined according to a flowchart. However, quantitative influence information is calculated from sunshine factor information such as feature shade information, contrast information, and halation information, and sunshine characteristic information, and the degree of influence of sunshine is determined based on this influence information. May be. For example, when the feature shade information is “partial shade”, the influence coefficient is set to “7”, when it is “shade”, the influence coefficient is set to “5”, and when it is “Hyuga”, the influence coefficient is set. Is “1”. Further, in the present embodiment, when the feature shade information is “shade”, the influence coefficient is “5” as the influence coefficient regarding the sunshine characteristic information, and when the sunshine characteristic information is “influenced by contrast”. In the case of “no influence”, the influence coefficient is set to “0”. When the feature shade information is “partial shade”, the influence coefficient regarding the sunshine characteristic information is set to “5” regardless of the sunshine characteristic information, and when the feature shade information is “sunshine”, the influence factor coefficient is related to the sunshine characteristic information. The influence coefficient is set to “0” regardless of the sunshine characteristic information. It should be noted that when the contrast is “small” according to the magnitude of the contrast, it is preferable to subtract “2” from the influence coefficient related to the sunshine characteristic information with “0” as the lower limit. As the influence coefficient of halation information, “15” is set only when the feature shade information is “hinata”, and “0” is set otherwise.

  Applying the above coefficients to the example of FIG. 10, the feature f1 that is the road marking of the pedestrian crossing and the feature f2 that is the road marking of the stop line have the feature shade information “Hyuga” and the halation information “None”. When the halation information is “None”, the influence coefficient is “0”. Therefore, when the feature shade information is “Hyuga”, the influence coefficient is “1”, and the quantitative influence information is “0”. " For the feature f3 and the feature f4, which are road markings for the pedestrian crossing notice, the feature shade information is “shade”, the contrast information is “large”, and the sunshine characteristic information is “influence of insufficient contrast”. is there. Therefore, the influence coefficient “5” when the feature shade information is “shade” and the influence coefficient “5” of the sunshine characteristic information are multiplied by “5”, and the quantitative influence information becomes “25”. The feature shade information of the feature f6 that is the road marking of the arrow is “partial shade”, and the contrast information is “large”. Therefore, the influence coefficient “7” in the case of “partial shade” is multiplied by the influence coefficient “5” set regardless of the sunshine characteristic information, and the quantitative influence information becomes “35”. For example, this quantitative influence information is classified into group A when it is 0-14, group B when it is 15-24, and group C when it is 25 or more. As described above, it is possible to determine the degree of the influence of sunshine by quantitative calculation.

  As described above, in the example shown in FIG. 10, the four types of features f are classified into two groups, group A and group C. As described with reference to the flowchart of FIG. 6, here, group A is preferentially selected, and among the candidate features belonging to group A, the feature f having the highest standard recognition rate is selected as the target feature. Is done. For the feature f1 (pedestrian crossing) and the feature f2 (stop line) belonging to the group A, the feature f1 has a higher standard recognition rate. Therefore, in the example shown in FIG. The feature f1 (crosswalk) is selected as the object.

  FIG. 11 shows an example of selecting a feature for a sunshine factor different from FIG. The difference from FIG. 10 is contrast information and halation information. In the example of FIG. 11, the contrast information is “small” and the halation information is “large”, so the feature shade information is “hinata” rather than “shade” or “partial shade” feature. Is a more severe condition. The feature f1 (pedestrian crossing) and the feature f2 (stop line) whose feature shade information is “Hyuga” have “Large” halation information, so the degree of influence on image recognition is “Large” Classified into group C. On the other hand, for the feature f3 and the feature f4 (pedestrian crossing notice) whose feature shade information is “shade”, the sunshine characteristic information is “insufficient contrast”, but the contrast information is “small”. The degree of influence is “small” and is classified into group B. In addition, the feature f6 (arrow) whose feature shade information is “partial shade” has the influence of shade indicated as the sunshine characteristic information “none” and the contrast information is “small”. The degree of influence is “small” and is classified into group B.

  As described above, in the example shown in FIG. 11, the four types of features f are classified into two groups, group B and group C. As described with reference to the flowchart of FIG. 6, here, group B is preferentially selected, and among the candidate features belonging to group B, feature f having the highest standard recognition rate is selected as the target feature. Is done. For the features f3 and f4 belonging to the group B (the pedestrian crossing notice) and the feature f6 (arrow), the features f3 and f4 have a higher standard recognition rate. Therefore, in the example shown in FIG. Features f3 and f4 (notice of pedestrian crossing) are selected as target features. As described above, the feature f3 and the feature f4 can also constitute the feature set g1. Therefore, the feature set g1 may be selected as the target feature. Further, any one of the feature f3 and the feature f4, for example, the feature f4 on the front side in the traveling direction may be preferentially selected.

  FIG. 12 shows an example in which features are selected by a method different from that in FIG. 10 (FIGS. 5 and 6) according to the same sunshine factor as in FIG. Specifically, step # 31 to step # 36 and step # 48 in FIG. 5 are common, but in step # 41 to step # 45, the reference recognition rate is corrected to the estimated recognition rate. That is, the reference recognition rate is corrected to the estimated recognition rate according to the degree of influence on the image recognition, that is, the possibility that the actual recognition rate will be lower than the reference recognition rate. Therefore, classification into a plurality of groups as in step # 49 is not performed.

  As described above, the feature f1 (pedestrian crossing) and the feature f2 (stop line) whose influence on the image recognition is “none” are unlikely to decrease in actual recognition rate (reduction amount = 0). %)It is estimated to be. The estimated recognition rate is 70% for the feature f1 (pedestrian crossing) and 60% for the feature f2 (stop line), similarly to the reference recognition rate. On the other hand, the degree of influence of the features f3, f4 (crosswalk notice) on the image recognition is “large”, and it is estimated that the actual recognition rate may be reduced by 25%, for example. As a result, the estimated recognition rate is 65%. Similarly, it is estimated that the feature f6 (arrow) whose influence on image recognition is “large” may have a 35% decrease in the actual recognition rate. As a result, the estimated recognition rate is 15%. The target feature selection unit 58 selects a feature having the highest estimated recognition rate as the target feature among the candidate features. In the example shown in FIG. 12, a feature f1 (pedestrian crossing) with an estimated recognition rate of 70% is selected. In addition, when the feature shade information is “partial shade” and the contrast information is “small”, it is estimated that the actual recognition rate may be reduced by about 20%. It is preferable that an estimated value of the recognition rate decrease corresponding to each of the steps # 41 to # 45 of 5 is set. Note that an estimated value of a reduction in recognition rate may be set based on the quantitative influence information described above.

  In the above description, the case where the target features are selected by classification into a plurality of groups and the case where the target features are selected by correcting the reference recognition rate to the estimated recognition rate have been described. You may use both together, without selectively using one method. In other words, the actual recognition rate may be lower than the standard recognition rate, that is, it is classified into multiple groups based on the degree of influence on image recognition, and the standard recognition rate is corrected to the estimated recognition rate. Also good. Then, a feature having the highest estimated recognition rate among candidate features belonging to the group having the lowest degree of influence on image recognition may be selected as the target feature.

[Learn feature information including sunshine characteristic information (feature information learning system)]
As described above, in order to select a target feature in view of the sunshine factor, it is necessary to prepare feature information including sunshine characteristic information indicating the influence on the image recognition by the sunshine factor with respect to the feature f. Therefore, it is preferable to construct a feature information learning system that learns feature information that is information related to features that are targets of image recognition as described below. The feature information can be collected, for example, by running a vehicle dedicated to collecting information, but can also be collected in cooperation with the navigation system 1 including the feature recognition system 4 described above. It is. In the present embodiment, a feature information learning system that collects feature information including sunshine characteristic information in cooperation with the navigation system 1 including the feature recognition system 4 described above will be described as an example.

  As shown in FIG. 1, the feature information learning system 8 of the present embodiment includes a learning calculation unit 80 configured with a learning storage unit 83 as a core, a learning database 19, and a feature information learning unit 85. It is configured. The learning storage unit 83 is a functional unit that stores sunshine factor information including feature shade information and contrast information in the learning database 19 in association with the image recognition result of the feature by the image recognition unit. The feature information learning unit 85 generates sunshine characteristic information indicating the degree of influence on image recognition by sunshine including at least shade for each feature based on the information stored in the learning database. It is a functional part which stores as the information which comprises the feature information about each.

  The learning calculation unit 80 having the learning storage unit 83 as a core is preferably realized together with the feature recognition system 4 in a calculation device (and software that cooperates) as a core of the navigation system 1. On the other hand, the feature information learning unit 85 is preferably realized in an arithmetic device (and cooperating software) such as a management center that maintains the navigation database 11. The feature information learning unit 85 that is centrally managed in the management center generates sunshine characteristic information using a plurality of information from the learning storage unit 83 that cooperates with each navigation device 1 to generate feature information. Can do. It is preferable that the plurality of learning storage units 83 and the feature information learning unit 85 managed in an integrated manner are connected to each other via a wired or wireless network. Further, information may be transmitted from each learning storage unit 83 to the feature information learning unit 85 via a storage medium such as a memory card. Further, there may be a medium that mediates each learning storage unit 83 and the feature information learning unit 85. For example, the information may be transmitted via a vehicle dealer, a service center of a manufacturer of the navigation system, a service center of an application software manufacturer or distributor operating on a portable device, and the like. The latest feature information generated by the feature information learning unit 85 can be provided to the navigation database 11 of each navigation system 1 through a similar transmission path.

  FIG. 1 illustrates a system configuration when the feature information learning unit 85 is centrally managed in a management center or the like. Therefore, the case where the learning database 19B (19) is also centrally managed in the management center is illustrated. The learning calculation unit 80 includes a learning database 19A (19) for temporarily storing information before transmitting information to the learning database 19B. Needless to say, the feature information learning unit 85 is not necessarily centrally managed in this way, and may be provided in each navigation system 1. And in each navigation system 1, the feature information of each navigation database 11 may be updated. Of course, only one learning database 19 is required in this case.

  Hereinafter, the procedure of learning of feature information including sunshine characteristic information by the feature information learning system 8 will be described with reference to the flowcharts of FIGS. The feature information learning system 8 is similar to the self-location specifying system 2 described above. The captured image acquisition unit 61, the image recognition unit 63, the date and time information acquisition unit 55, the estimated position information acquisition unit 42, and the feature information. An acquisition unit 43, a contrast information acquisition unit 51, and a feature shade information calculation unit 53 are provided. These functional units are as described above, but will be briefly described again in order to clarify the functions when viewed as the feature information learning system 8.

  The captured image acquisition unit 61 is a functional unit that acquires a captured image captured by the camera 16 (imaging device). The captured image includes a feature to be learned, that is, a learned feature. The image recognition unit 63 is a functional unit that performs image recognition of a feature included in a captured image. In the above description, it has been described that the image recognition unit 63 that functions as a part of the self-localization system 2 recognizes only the selected target feature. However, the image recognition unit 63 functioning as a part of the feature information learning system 8 recognizes a plurality of learning features. As will be described in detail later, the image recognition unit 63 performs image recognition of one target feature selected from among a plurality of candidate features, and when the image recognition of the target feature is successful, Image recognition of a learning feature that is a candidate feature different from the target feature is performed. The date information acquisition unit 41 is a functional unit that acquires date information indicating the current date. The estimated position information acquisition unit 42 is a functional unit that acquires estimated position information indicating its own position, in particular, the estimated position of the camera 16 here.

  The feature information acquisition unit 42 is a functional unit that acquires feature information about candidate features existing within the search range SP (predetermined range). Here, the feature information acquisition unit 42 acquires feature information about a plurality of candidate features (learning features) existing along the movement path of the camera 16. Of course, the range may be limited, the search range SP may be determined, and feature information of candidate features existing along the movement path of the camera 16 may be acquired within the search range SP. You may acquire the feature information of the candidate feature along the said movement path | route sequentially, without defining. In addition, the feature information may be learned using a captured image by a fixed imaging device. In this case, since the position of the imaging device is fixed, there is no need to consider a moving route or the like. The contrast information acquisition unit 51 is a functional unit that acquires contrast information indicating the contrast between the shade and the sun. Further, the feature / shade information calculation unit 53 can distinguish at least a shade state in which all the features are included in the shade, a sunlight state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. It is a functional unit that calculates feature shade information that is information indicating the positional relationship between the feature and shade for each feature.

  By these function units, as shown in steps # 71 to # 75 in FIG. 13, estimated position information, feature information, captured images, date and time information, and sunshine factor information are acquired. In these steps, Step # 71 is Step # 01 in FIG. 3, Step # 72 is Step # 02 in FIG. 3, Step # 73 is Step # 07 in FIG. 3, and Step # 74 is Step # in FIG. In step 09, step # 75 corresponds to step # 20 of FIG. 3 and FIG. As described above, the feature selection system 5 uses this information to select one target feature from the search range SP. The feature learning system 8 stores, for example, a captured image within the search range SP while the feature selection system 5 selects a target feature and the image processing system 6 performs image recognition of the target feature. The temporary storage unit 81 temporarily stores the information in association with the position information. At this time, the sunshine factor information may also be associated and temporarily stored in the temporary storage unit 81. Here, the search range SP is temporarily stored. However, as described above, the feature learning may be performed regardless of the search range SP. Therefore, it is preferable that the temporary storage unit 81 is configured as a functional unit that accumulates captured images over a predetermined period and temporarily stores them in association with estimated position information.

  When the image recognition of the target feature by the image processing system 6, that is, the image recognition of the target feature by the image recognition unit 63 is completed, it is determined whether or not the image recognition of the target feature is successful (# 76 in FIG. 13). : Target feature recognition result determination step / function). If the image recognition of the target feature is not successful, the feature information learning system 8 ends the learning process. On the other hand, when the image recognition of the target feature is successful, the candidate position different from the target feature exists on the captured image by correcting the estimated position information to the high-accuracy position information. Since the position can be predicted with high accuracy, the learning of the feature information is continued.

  That is, the image recognizing unit 63 recognizes an image of one selected target feature, corrects the estimated position information to high-accuracy position information, and is included in the captured image and is a candidate different from the target feature. Image recognition of learning features that are features is performed. Specifically, when the image recognition of the target feature is successful, the image recognition unit 63 estimates the estimated position stored in the temporary storage unit 81 based on the position information included in the feature information about the target feature. Correct the information. That is, the index information based on the position information in the temporary storage unit 81 is changed. Then, the image recognition unit 63 performs image recognition of the learned feature based on the captured image read from the temporary storage unit 81 based on the corrected estimated position information (high-accuracy position information) and the candidate feature information. Do. Since the index information in the temporary storage unit 81 has been changed to high-accuracy position information, the learning feature is based on the position information of each feature defined in the candidate feature information that is the feature information about the candidate feature. It is possible to accurately obtain a captured image that is estimated to contain the. Such processing continued after step # 76 will be described below.

  If it is determined in step # 76 that image recognition of the target feature has succeeded, it is next determined whether or not there is a candidate feature other than the target feature, that is, a feature that becomes a learning feature (#). 77: Learning feature existence confirmation step). When there is no feature to be a learning feature, the feature information learning system 8 naturally ends the learning process. On the other hand, when there is at least one feature to be a learning feature, high-accuracy position information is acquired (# 81: high-accuracy position information acquisition step). The high-accuracy position information may be calculated by the position information correction unit 32 of the positioning system 3 or may be calculated by the image recognition unit 63.

  Subsequently, the image recognition unit 63 performs image recognition of the learning feature based on the photographed image read from the temporary storage unit 81 based on the high-precision position information and the candidate feature information (# 82: learning). Feature image recognition step). When the image recognition by the image recognition unit 63 is completed, the learning storage unit 83 stores the sunshine factor information and the image recognition result of the learning feature in association with each other in the learning database 19 (# 83: learning storage step). The learning feature is a feature that is not selected as the target feature among the candidate features, and is therefore a feature that is more likely to be unsuitable for image recognition than the target feature. However, since image recognition is performed using the captured image acquired from the billion part 81 based on the high-accuracy position information, the degree of matching in pattern matching of the feature is calculated with high accuracy. For this reason, it is preferable to store not only the success / failure of recognition but also the information on the fitness in the learning storage unit 83 as a recognition result. Thereby, the feature information learning unit 85 can accurately generate the influence of sunshine, that is, the sunshine characteristic information.

  The degree of conformity is a value that quantitatively indicates the ratio of the feature amount extracted from the photographed image and the matching amount in matching with the dictionary 64. If the feature quantity of the extracted feature and the dictionary 64 completely match, the fitness is 100%. For example, when the fitness is a predetermined fitness threshold (for example, 80%) or more, it is determined that the image recognition is successful. The reference recognition rate is the ratio of the number of times that the fitness is equal to or higher than the fitness threshold when a feature is image-recognized multiple times for different captured images. By storing the fitness level, the feature information learning unit 85 to be described later determines the degree of the influence on the image recognition depending on how much the fitness level falls below the fitness threshold value as a result of the image recognition. Can be calculated quantitatively. Therefore, it is possible to calculate more accurate sunshine characteristic information. In the case where the camera 16 is a fixed photographing device, the accuracy of the position information may be high without obtaining high-accuracy position information by image recognition of the target feature. In such a case, image recognition of the target feature is unnecessary, and the feature within the search range SP may be recognized and learned.

  As described above, the sunshine factor information and the image recognition result of the learned feature are stored in the learning database 19 in association with each other. The feature information learning unit 85 generates sunshine characteristic information based on the information accumulated in the learning database 19 and stores it as information constituting the feature information. In order to generate sunshine characteristic information, it is required that a predetermined amount or more of information is accumulated in the learning database 19. Therefore, the sunshine characteristic information is generated based on the information accumulated in the learning database 19 by automatically determining that the information has reached a predetermined amount, or at predetermined intervals, or by an artificial instruction. The As an example, it is preferable that the sunshine characteristic information is calculated by statistically analyzing the fitness degrees accumulated as described above. That is, for each of the sunshine factor information, the fitness is classified and accumulated, and depending on the difference between the accumulated fitness and the fitness threshold or the average fitness or target fitness of the feature. Thus, it is preferable that the degree of influence on image recognition is calculated to generate sunshine characteristic information. As shown in FIG. 14, the presence / absence of a navigation database generation instruction is determined (# 91), and when there is a generation instruction, sunshine characteristic information is generated (# 92). Then, the generated sunshine characteristic information is incorporated into the feature information as information constituting the feature information, and the feature information is updated. Learning is completed by storing the updated feature information in the navigation database 11 (# 93).

[Other Embodiments]
Hereinafter, other embodiments will be described. Note that the configuration of each embodiment described below is not limited to being applied independently, and can be applied in combination with the configuration of other embodiments as long as no contradiction arises.

(1) In the above embodiment, for example, as described with reference to FIG. 5, the influence on the image recognition by the sunshine is determined using the contrast information, the halation information, and the feature shade information as the sunshine factor information. An example to show. However, for example, without using halation information, the influence of sunshine on image recognition may be determined using contrast information and feature shade information as sunshine factor information. Those skilled in the art can easily realize the processing by excluding the processing related to halation information from the flowchart illustrated in FIG.

(2) In the above embodiment, the case where the feature shade information has information that can distinguish between “shade”, “hinata”, and “partial shade” is exemplified, but the present invention is not limited to this. . For example, it may be information indicating only whether it is a partial shade. In the case of shade and sun, the brightness of the captured image is uniform, but in the case of partial shade, the brightness changes greatly. Therefore, when the contrast of the captured image is large, the influence on the image recognition becomes large. Therefore, even if the feature shade information is information indicating only whether or not the feature shade information is partial shade, the effect in feature selection is great. As an example, the target feature selection unit 58 may select a target feature by lowering the priority of candidate features whose feature shade information indicates partial shade. A person skilled in the art can easily implement the process related to the feature shade information in the flowchart illustrated in FIG.

(3) In the above-described embodiment, as illustrated in FIGS. 5 and 9, the example in which the influence on the image recognition due to the sunshine is determined using the sunshine characteristic information included in the feature information has been described. However, the influence of sunshine on image recognition may be determined without considering sunshine characteristic information. Specifically, after step # 32 in FIG. 5, the process may proceed to step # 34 without passing through step # 33.

(4) In the above-described embodiment, the navigation database 11 is configured to include the three-dimensional object database 11b that stores the three-dimensional object information including at least the position information and the shape information of the three-dimensional object that is a factor for forming a shade. Indicated. Then, the three-dimensional object information about a plurality of three-dimensional objects existing within the search range SP (predetermined range) is acquired by the three-dimensional object information acquisition unit 44, and the feature shade information calculation unit 53 uses the three-dimensional object information and date / time information. The form which calculates feature shade information based on the position of the sun based on was illustrated. However, the feature selection system 4 and the feature information learning system 8 may calculate and acquire the feature shade information without using the three-dimensional object information read from the three-dimensional object database 11b.

  For example, a sunshine state detection unit 55 may be provided that performs image processing such as making a captured image into a mosaic and detects a sunshine state based on a luminance distribution or change in the captured image. Further, as shown in the lower part of FIG. 8, an illuminance sensor 17 is mounted on the vehicle 100 or the camera 16, and the sunshine state detection unit is based on the illuminance detected by the illuminance sensor 17 as the vehicle 100 or the camera 16 moves. 55 may detect a change in illuminance. In FIG. 8, symbol L indicates that the luminance is high and bright, and symbol D indicates that the luminance is low and dark. Thereby, it can be determined whether each feature exists in a bright area or a dark area. That is, according to the position information included in the feature information, if the feature is present in a bright area, the feature shade information is calculated as “Hyuga”. If the feature is present in a dark area, the feature shade information is calculated as “shade”. Further, when the position information included in the feature information changes from a bright region to a dark region or from a dark region to a bright region in the middle of the region where the feature exists, “partial shade” is calculated.

(5) In the above embodiment, for example, estimated position information indicating the estimated position of the vehicle 100 or the camera 16 (imaging device) is acquired as information on the own position, and the candidate feature exists based on the estimated position information. The procedure for setting a predetermined range (search range SP) as a range is illustrated. However, without being limited to this procedure, the search range SP may be set based on the position where the course is changed or an arbitrary position.

(6) In the above embodiment, an example in which the reference recognition rate is used when selecting a target feature from a plurality of candidate features has been described. However, the target feature may be selected without using the reference recognition rate. For example, as shown in FIGS. 5, 10, and 11, when classified into a plurality of groups, a feature close to the photographing apparatus is selected as a target feature among the features belonging to the selected group. May be. Alternatively, the priority order may be determined in advance according to the type of the feature, and the target feature may be selected according to the priority order from the features belonging to the selected group. As shown in FIG. 12, when calculating the degree of reduction in the recognition rate, it is preferable to select the target feature by giving priority to the feature with a small reduction in the recognition rate. When there are a plurality of features having the same amount of decrease in recognition rate, it is possible to set priorities in the same manner as when classified into groups.

(7) In the above embodiment, the navigation system has been described as an application example. However, the feature selection system can naturally be used in applications other than the navigation system. For example, in an image recognition system for recognizing an administrative division from a photograph or an image recognition system for recognizing an empty state of a parking lot, it can be used as an application for selecting a feature used for recognition.

  As described above, the present invention can be applied to a feature selection system that selects a target feature to be subjected to image recognition from a plurality of features whose presence is known in advance. In addition, the present invention can be applied to a feature information learning system that learns feature information that is information about a feature that is a target of image recognition. The present invention can also be applied to an image recognition system that recognizes an image of a feature selected by such a feature selection system. Furthermore, the present invention can also be applied to a self-position specifying system that specifies a position based on the recognition result of such an image recognition system. Furthermore, the present invention can also be applied to a navigation system that guides a route or the like based on the own position specified by such an own position specifying system.

5: feature selection system 8: feature information learning system 11f: feature database 11b: three-dimensional object database 19, 19A, 19B: learning database 41: date and time information acquisition unit 42: estimated position information acquisition unit 43: feature information acquisition Unit 44: three-dimensional object information acquisition unit 51: contrast information acquisition unit 52: halation information acquisition unit 53: feature shade information calculation unit 55: sunshine state detection unit 58: target feature selection unit 61: captured image acquisition unit 63: image Recognition unit 80: Learning calculation unit 81: Temporary storage unit 83: Learning storage unit 85: Feature information learning unit 200, 201, 202, 203: Solid object f, f1 to f6: Feature SP: Search range (predetermined range)
SH, SH1, SH2, SH3: Shadow

Claims (5)

A feature information learning system that learns feature information that is information about a feature that is subject to image recognition,
A captured image acquisition unit for acquiring captured images;
An image recognition unit for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation unit for calculating feature / shade information, which is information indicating a positional relationship between the feature and the shade,
A contrast information acquisition unit that acquires contrast information indicating contrast between the shade and the sun;
A learning storage unit that stores sunshine factor information including the feature shade information and the contrast information and an image recognition result of the feature by the image recognition unit in association with each other in a learning database;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning unit that stores information as constituent information;
Feature information learning system with A photographing device for photographing the photographed image;
An estimated position information acquisition unit that acquires estimated position information indicating an estimated position of the imaging device;
A temporary storage unit that accumulates the captured image over a predetermined period and temporarily stores the image in association with the estimated position information;
A candidate feature that is connected to a feature database that stores the feature information including at least the position information of the feature and that is the feature information of a plurality of candidate features that exist along the movement path of the imaging device A feature information acquisition unit for acquiring information,
The image recognition unit performs image recognition of one target feature selected from the plurality of candidate features, and when image recognition of the target feature is successful, The estimated position information stored in the temporary storage unit is corrected based on the position information included in the feature information, and the captured image is read from the temporary storage unit based on the corrected estimated position information. , Based on the candidate feature information, image recognition of the learning feature that is the candidate feature different from the target feature included in the captured image,
The feature information learning system according to claim 1, wherein the learning storage unit stores the result of the image recognition and the sunshine factor information in a learning database in association with each other. A sunshine state detection unit that detects a sunshine state at each position in association with path position information indicating a position along the movement path of the imaging device;
The feature shade information calculation unit obtains the feature shade information for each of the features based on the detection result of the sunshine state detection unit and the position information of the feature included in the feature information. Operate,
The image recognition unit is included in the captured image stored in the temporary storage unit based on position information included in the feature information about the learned feature and the corrected estimated position information. The feature information learning system according to claim 2, wherein image recognition of the learned feature is performed. A feature information learning program for learning feature information, which is information about a feature to be image-recognized,
A captured image acquisition function for acquiring captured images;
An image recognition function for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation function for calculating feature / shade information, which is information indicating the positional relationship between the feature and the shade,
A contrast information acquisition function for acquiring contrast information indicating contrast between the shade and the sun;
A learning storage function for associating and storing in the learning database the sunshine factor information including the feature shade information and the contrast information, and the image recognition result of the feature realized by the image recognition function;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning function to store as constituent information;
Is a feature information learning program that makes computers realize. A feature information learning method for learning feature information, which is information about a feature to be image-recognized,
A captured image acquisition step of acquiring a captured image;
An image recognition step for performing image recognition of the feature included in the captured image;
It is information that can distinguish at least a shade state in which all the features are included in the shade, a sunny state in which all the features are included in the shade, and a partial shade state in which some of the features are included in the shade. A feature / shade information calculation step for calculating feature / shade information, which is information indicating the positional relationship between the feature and the shade,
A contrast information acquisition step of acquiring contrast information indicating contrast between the shade and the sun;
A learning storage step of associating and storing a sunshine factor information including the feature shade information and the contrast information in a learning database in association with an image recognition result of the feature by the image recognition step;
Based on the information stored in the learning database, for each of the features, sunshine characteristic information indicating a degree of influence on image recognition by sunshine is generated, and the feature information about each of the features is A feature information learning step to store as constituent information;
A feature information learning method comprising:

Images