JP2008287379A - Road sign data input system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce time and work for constructing database by extracting sign information (a classification and a position) from road image data photographed by a measuring vehicle. <P>SOLUTION: By starting functions to recognize a sign and to calculate the position to be surveyed from a menu of an interactive recognition and edit system, and performing recognition processing before editing to display an evaluation of a sign candidate to an icon size on a map, a place where a sign is recognized can be easily confirmed and an interface configuration that is not affected by erroneous recognition by survey calculation can reduce the number of processes of an operator. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to the field of construction of road sign databases required for car navigation systems and ITS (Intelligent Transfer System), and in particular, for extracting sign information from road images captured from in-vehicle cameras, confirmation of automatic recognition results And an interface technology for reducing the load of the process of correcting the erroneous recognition result.

  Road signs are originally installed on roads and intersections, but a system that supports the driver's safe driving and accident prevention by displaying information equivalent to that road sign on the in-vehicle terminal and alerting by voice The demand is growing. For this purpose, it is necessary to acquire sign information that associates the type of road sign with its location.

  Road signs installed on roads and intersections include those that present traffic regulations such as vehicle speed restrictions, entry prohibitions, and parking prohibitions, and those that present guidance information such as intersection names and destination names. Among them, the traffic signs that show the former traffic regulations include standard instruction information for instructing traffic regulations, such as speed limit (speed value in red circle), parking prohibition (white line in red circle), Since the purpose is to indicate a direction of travel (arrow) or the like, the number of signs is limited to about 60 to 80 patterns. Conventionally, a number of technologies for automatically recognizing the type of sign using a photographed image of an in-vehicle still image camera or video camera for a road sign indicating such traffic regulation have been announced. For example, in Document 1, since road signs are described in specific colors (red, blue, yellow, green, black, white) with specific patterns, color separation of specific colors and specific shapes (red edges) Image recognition technology focusing on (circle) has been developed. Reference 2 proposes a method for optimally determining a color separation cluster threshold so that the influence of solar illuminance does not act on color separation recognition. Further, Document 3 has applied for a recognition speed-up method for a partial area provided with a slit area for recognizing a sign from an on-vehicle camera image at high speed.

  If the real-time sign information acquisition method using these image recognition technologies is viewed from the viewpoint of in-vehicle systems, it is difficult for the recognition rate to reach a practical level (for example, 95% or more), and some human judgment or correction is required. Therefore, it is not necessarily put into practical use as an in-vehicle system. Instead, the captured image data of the road and the positioning data are temporarily stored in the storage of the measurement vehicle, taken back to the data input center, and built as a sign database by batch input editing, and the database is stored in the car navigation system. The method of providing From the point of view of constructing such a sign database, it is similar in interface to using a manual input method by an operator from the beginning, so a recognition function is incorporated as part of the editing function for database construction. Many of these systems are called interactive recognition editing systems.

Uchimura, et al. 2: Extraction and recognition of circular road signs in road scene color images, IEICE Transactions AVol.J81-A-4, 1998 H. et al. 1: Recognition of road signs in color images, IEICE Transactions Vol.J87-D-II, No. 12, 2004 Links Image Systems Division: How to use HALCON, Links Publishing Division, 2004 JP 2000-293670 A JP 2003-123197 A

  Road signs installed on roads and intersections have been repeatedly lost due to new establishments associated with the opening of new roads, relocations associated with road improvements, accidents, and natural disasters. For new roads, the location and type of road signs are described in the road design drawings, so the road sign database should be created by extracting the sign information from the new road design drawings delivered to the Road Administration Bureau, etc. However, it is judged that it is difficult to construct a database from the design drawing because the change of the road sign accompanying the road improvement is not necessarily delivered as a comprehensive design drawing. In case of loss due to an accident or natural disaster, there is no guarantee that the same kind of sign will be re-installed at the original position. Therefore, in order to create the road sign database, it is necessary to actually investigate the road signs installed on the road and obtain the new or changed sign information (type and position information of the signs).

  In order to reduce this survey work as much as possible, a measurement vehicle equipped with a video camera and a high-accuracy position sensor was constructed, and by accumulation of data by road traveling of the vehicle and video shooting / position measurement and analysis of the shooting data, A method for obtaining road sign information has been developed. Most of the work to construct a sign database is because there are many types of road signs and their sizes are different, so the operator can investigate the shooting data, determine the type, and position measurement data (often installed in measurement vehicles). (Such as measurement using a stereo camera). Since such an operator's operation cannot be performed in real time in the measurement vehicle, the data storage step by the measurement vehicle and the editing step by the operator are often performed separately. It is not an exaggeration to say that most of the construction cost of the sign database is the editing process cost of the operator, and the reduction of man-hours has become a problem.

  The most effective way to reduce the number of man-hours is the introduction of automatic sign recognition for photographing data, and the key to reducing man-hours in that case is the accuracy of sign recognition. Various image recognition algorithms have been proposed for the sign recognition, and the literatures 1, 2, and 3 already described describe in detail algorithms related to the sign image recognition method. However, such sign recognition always includes a recognition miss and a recognition error. Therefore, even if a high-accuracy image recognition function is used, the recognition rate does not reach 100%, so an operator must confirm the recognition result. In addition, the operation is redundant, and it is necessary to find out a scene where a sign appears from a long captured image sequence, and to check whether each recognized result is correct.

  In the imaging data provided from the measurement vehicle, even if it is one sign, it appears in an image sequence picked up at a plurality of observation points, so that a plurality of recognition results can be obtained for the same sign. . Moreover, the recognition accuracy of the sign depends on the size of the sign appearing in the image sequence and the influence of sunlight or shadow. Considering the operator operation for inputting the sign data under such circumstances, it takes a long time and man-hour to check in which part of the image sequence the sign appears before confirming the recognition result itself. It is expected that. In addition, an interface method that arranges and displays the recognition result at the survey position on the map and confirms the quality of the result is also conceivable, but if misrecognition occurs, not only find the place, There arises a problem that requires a great number of man-hours for grasping the position and cause of the image causing the erroneous recognition and correcting the recognition result so that there is no contradiction.

  The present invention has been devised to solve such a problem, and the label recognition process for the image data sequence photographed by the measurement vehicle is divided into two processes, that is, the sign appearance prediction recognition and the sign identification recognition. The operator proposes a method of starting from a dialog operation using the result of the preceding sign appearance prediction recognition and operating the subsequent sign identification recognition to confirm the result. As described above, since an image sequence in which a sign appears at a plurality of shooting positions is provided for one sign, a plurality of recognition results can be obtained for the same sign. Moreover, since the recognition accuracy depends on the size of the sign appearing in the image and the photographing conditions such as sunlight, the possibility of different recognition results for the same sign increases. Therefore, by performing sign appearance prediction recognition mainly composed of sign feature extraction, a part where a sign appears in a long image sequence is distinguished from a part where it does not. In such a case, the ranking is performed according to the evaluation value of the feature value, the operator operation is started from the position with the highest rank, and the recognition result is confirmed by starting the label identification operation from the dialogue menu. And

  Furthermore, as an interface for displaying the result of the sign appearance prediction recognition to the operator, the predicted appearance rate is expressed by a symbol having a size corresponding to the ranking of the evaluation value of the feature amount (for example, a circle having a radius corresponding to the evaluation value). Etc.) and the display color so that the operator can easily confirm the position of the sign photographing where the highest recognition rate can be expected. In addition, in order to increase the efficiency of marker identification recognition performed continuously after the confirmation of the prediction recognition, a list display is made so that the predicted marker appearance position, its rank value, and the result of the marker identification recognition can be understood correspondingly. Provide an interface that facilitates related operations.

  With such an interface method, it is possible to know in advance the position where the sign appears on the image for the image series from the in-vehicle camera of the measurement vehicle. It is only necessary to perform the marker identification process while paying attention to the part having a high value. In addition to the reduction in the editing efficiency time, the recognition processing time can also be reduced. Therefore, the knowledge data process is expected to greatly reduce the data input cost.

  FIG. 1 shows the overall configuration of a sign data input system which is an embodiment for carrying out the present invention. This system has the following configuration. First, a central processing unit (1) that performs recognition and correction of marker data and database registration management, etc., and a pointing device (2) composed of a keyboard and a mouse for instructing operator operations, and the operator It is assumed that a display device (3) for displaying and confirming the progress and result of the operation is connected to the central processing unit. The central processing unit is subject to processing of various measurement data (4) provided from the measuring vehicle, and the processing output is output data (5).

  Next, the hardware configuration of the central processing unit may be the configuration of a general computer equipped with a memory and storage centered on the CPU of a personal computer or workstation, but the internal software configuration is as follows: It is assumed that the processing part is configured. A sign recognition unit (10) that automatically recognizes the sign part existing in the screen for the image from the in-vehicle camera of the measurement vehicle, and a survey calculation part that calculates the sign position using a plurality of these recognized signs (11) Interactive editing input unit (12) for interactively editing the processing target area using the display screen and the pointing device and applying these processes or correcting the defective part, the above process Database management unit (13) for registering and managing the sign data obtained from the management and processing of images provided by the measurement vehicle required by the system, the database (14), and data for inputting measurement vehicle measurement data to the database A DB input conversion unit (15) that performs conversion, and a DB output unit (16) that performs data conversion for outputting the contents of the database to the external environment.

  The central processing unit (1) is a general-purpose computer and performs processing such as DB input, sign recognition, survey calculation, interactive editing input, database management, DB output, and the like. If this device has a function to communicate with the outside, parts other than the processing performed by the interactive editing input unit (10, 11, 12, 13, 14, 15, 16) are performed by a server at a remote location. You may make it perform. In that case, it is possible to speed up the system response in the dialog editing processing unit by performing processing with a high-performance computer. Further, by multiplexing the database management for a part of the data used by the interactive edit input unit with the server unit, a higher processing speed can be obtained, but these system configurations have the above two formats. It is not limited to.

  A series of flow from the sign recognition to the sign data registration in the database through interactive editing input in the system configuration will be described with reference to FIG. The sign recognition unit (10) first converts a color signal from an RGB signal system to an HSI signal system close to human color sense by color signal conversion processing (110). As shown in Non-Patent Document 2, the conversion method is based on R (Red), G (Green), and B (Blue) signals, and the hue (Hue; difference in color), saturation (Saturation; color It is to be converted to a signal system of density) and lightness (Intensity), and detailed conversion formulas are as described in the literature, so they are omitted here. Next, color components (red, blue, green, yellow, white, black) used in signs are extracted by threshold processing with different bandwidths according to brightness by color recognition and noise removal processing (110). At the same time, noise components attached to the periphery of the area including the color component are removed. Using the image components constituting the sign image thus obtained, candidate area extraction processing (120) is performed for extracting the area forming the sign as a circumscribed rectangle. It is assumed that a figure that forms a circumscribed rectangle frame as a marker candidate region is superimposed and displayed in a conspicuous color (for example, yellow) on the recognition target image in the region where the marker candidate can be extracted as the circumscribed rectangle.

  The above series of processing from 100 to 120 is performed in batches as the preprocessing of the subsequent series of interactive recognition correction processing (130 to 170), or only as much as necessary before the following repetitive processing. Designated by the operator. The following interactive recognition editing process by the operator is repeated until the processing target ends. When this repetitive processing is performed, it is performed on a window that displays a map or video as described below as a background. First, an operator observes an image of one scene provided from a measurement vehicle as a minimum unit to be processed, and checks whether or not a circumscribed rectangle of the marker candidate image extracted in the previous stage is displayed. As a result of judging whether there is a circumscribed rectangle indicating the candidate image of the sign (130), if it is presented, that part is subjected to the sign identification process (140) to recognize the detailed object of the sign To do. Otherwise, interactive editing / correction processing (150) is performed. The process is roughly divided into two editing / correction processes. One is a case where a sign candidate is not extracted in the previous candidate area extraction process even though there is a sign in the scene provided by the measurement vehicle, and the other is an erroneous target whose candidate area is different from the sign Is extracted. In the case of the former error, the circumscribing rectangle is designated by the operator's manual operation and the type of the sign is input. In the case of the latter error, the circumscribed rectangle extracted by the operator's operation is deleted, and processing that does not become a marker candidate is performed. The label position surveying calculation process (160) is performed on a series of images (two scenes or more) in which the label candidates are identified as specific labels by the processing so far. Then, registration processing (170) in the marker DB is performed in a state where the identification information of these markers and the position information are associated with each other.

  Next, a detailed flow of color recognition and noise removal processing will be described with reference to FIG. The repetition (111) corresponding to the number of combinations (three times in this case) using two components among the three HIS signals is applied to the following processing system. In this case, as shown in FIG. 4, when the horizontal axis represents lightness (Intensity) and the vertical axis represents saturation (Saturation), binarization is performed with a threshold having a saturation bandwidth corresponding to the lightness width. The method will be described. Focusing on [Ii, Ii + 1], the i-th width of the lightness axis divided by n equally spaced widths, the lower threshold of saturation corresponding to the lightness width is BTi, and the upper threshold is UTi And At this time, the image component belonging to the range of the saturation bandwidth [BTi, UTi] is set to 1, and the other image components are set to 0 (113). By repeating this threshold processing with a bandwidth n times from i = 1 to n, it is possible to extract an image region of a color component having a saturation with a specific bandwidth corresponding to the lightness (114). The newly extracted i-th region component is connected to the region so as to be the same as the region component extracted in the preceding i-1 (115). However, if the size of the i-th extracted region component is below a specific value, it is removed as a noise component (116). If the extracted area has a range that overlaps with the area extracted up to the previous stage, processing is performed to combine the overlapping areas by OR operation (117). However, the i-th repetition number and the band threshold number are held as characteristic values in the respective overlapping areas before the OR operation, so that it is possible to determine the color afterwards. By performing binarization based on the bandwidth saturation according to the lightness as described above, it is possible to perform stable color recognition that is less affected by changes in illuminance due to weather or the like and shadows. Note that the band-shaped threshold data relating to the saturation corresponding to each lightness value used here is determined in advance by an investigation process for sample data obtained from similar shooting conditions before performing the recognition process. . In addition, the threshold value used as the noise component is also determined by an investigation process for these sample data. The process so far is a stage where a silhouette for cutting out the sign area has been created. In the subsequent processes, the shape of each color in the sign area is determined and the sign is identified by a combination thereof. As a method of evaluating the likelihood of a sign, the silhouette extraction process evaluation up to the previous stage and the evaluation of the sign identification process should be added together, and the ratio of the evaluation value is determined in a fluid manner.

Next, the flow of processing for extracting candidate regions for marking from image components recognized up to the upper stage will be described with reference to FIG. First, assuming that M areas have been extracted by the color recognition process up to the previous stage in one scene to be processed, the following process is repeated for the number of areas (121). In order to accurately extract the outline of the sign as a whole, the inside of the sign is checked for the presence of a black component surrounded by white, which constitutes a small character or symbol, and the existence range is narrowed down. The symbol area is not subjected to the following label area extraction processing (123). From the comparison of the size of each individual area, a large area above a certain level or a minute area below a certain level is removed as a noise component, and the floating island state image missing in the remaining area A process for compensating for the scratch component of the outline image is added (123). A circumscribed rectangle is calculated for the marker candidate region remaining in the previous processing (124). Based on the features such as the area of the circumscribed rectangle and the aspect ratio, processing for further limiting the region that is a candidate for labeling is performed (125). Constituent colors and shape characteristics are obtained for the obtained image inside the circumscribed rectangle, and an evaluation value of the marker candidate region is obtained (126). As a calculation method of the shape feature at this stage, the processing speed is increased by obtaining the aspect ratio of the circumscribed rectangle of the region cut out for each color component. The recognition result obtained up to this stage is a candidate image area that forms a sign, which is equivalent to extracting a silhouette image area that forms the outline of the sign. Further, the evaluation value of the marker area obtained at this stage is used as a first evaluation value as a criterion for selecting a processing target of the interactive recognition editing system described later.
A flow of processing for identifying a sign for an image inside the sign candidate region extracted up to the previous stage will be described with reference to FIG. In order to identify the type of the sign by determining the pattern of the image inside the sign candidate area, the feature of the image in the sign candidate area is calculated, and the color and shape combination dictionary as shown in FIG. 7 is referred to. An evaluation value as a label is obtained (133). As an example of the calculation method of the feature evaluation value, if the individual feature evaluation value for the marker i such as each component color and shape feature is f _ij , the feature evaluation value F _i of the marker i is evaluated for each feature f _ij . The weight value w _j is combined and calculated by the following formula 1.

但しウエイト値wjの具体的な設定方法の例としては、速度規制や駐車禁止等の標識に必ず含まれる赤色の円弧形状成分を重視する意味で、構成色のウエイト値wjを他の緑色の0.4や黄色の0.3よりも大きな0.8にして色特徴が大きく出るようにする外、形状特徴の円弧に相当する項目のwjを0.7にして他の矢印の0.5や文字の0.3よりも大きくして形状特徴が大きく反映されるように設定する。

However, as an example of a specific method of setting the weight value wj, the weight value wj of the component color is set to 0.4 for other green colors in order to emphasize the red arc shape component that is always included in the signs such as speed regulation and parking prohibition. In addition to making the color feature appear larger by setting it to 0.8, which is larger than 0.3 for yellow and yellow, the shape feature is made by setting wj of the item corresponding to the arc of the shape feature to 0.7 and larger than 0.5 for other arrows and 0.3 for characters Set so that is greatly reflected.

  It is determined whether or not the evaluation value is equal to or greater than a certain value for each type of sign (134), and if it is greater than or equal to a certain value, it is registered in the candidate list as the first sign candidate value (135). Further, even when the value exceeds a certain value for each other label type, it is registered in the candidate list as the second label candidate value. This is repeated for the number of marker types and for the rear of the previous stage of narrowing down (131) (132) a. As a method for calculating the shape feature for the image inside the marker candidate region performed in the processing system of (133), any means for quantifying the shape feature such as a rectangle, a circle, and a triangle can be used. For example, a basic area-to-perimeter ratio calculation or a calculation value using a shape analysis library as described in Document 3 may be used. An evaluation value for identifying a sign obtained in the above process is defined as a second evaluation value, and used as a selection criterion for a processing target of the interactive recognition editing system described later.

  By the above procedure, the sign recognition for the road photographing image provided from the measurement vehicle is realized. An embodiment of a road sign input interface using the interactive recognition editing system according to the present invention will be described below.

  First, touching on the hardware configuration for executing the processing of the present embodiment, as already described in FIG. 1, the display device (2) and the pointing device (3) are connected to the central processing unit (1). . Focusing on the interactive recognition editing unit (12) in the central processing unit having this configuration, the configuration of the screen interface unit for the processing will be described with reference to FIG. From the left side of the screen configuration, the operation control unit (210) for operations such as data input / output and display / recognition / surveying, main display screen (220) for displaying a large map or image to be processed, scene Sub-display screen (230) for displaying maps and images as subordinates in order to grasp the context and position on the map, candidates for signing recognition candidates in order of high evaluation and candidates for confirmation input Display screen (240), position information display screen that displays detailed numerical information such as latitude, longitude, height, etc. related to the position indicated on the main screen, marker candidates recognized according to the order (series) specified on the main screen It consists of a recognition sequence screen (260) showing the type of the icon. Next, the flow of a series of processes from searching for the image DB from the database (14), applying the recognition editing process to registering it in the feature DB will be described using each of these screens.

  From the operation control screen (210), the operator searches the database for the captured image data to be processed, the travel trajectory of the measurement vehicle, and the background map, and displays them on the main display screen 220 and the sub display screen 230. As shown in FIG. 8, the display form in this case displays the travel locus of the measurement vehicle with a line with a map in the background, and the position taken by the in-vehicle camera on the map as a circle icon. On the secondary display screen, the scene at the shooting position designated by the pointing device and the scene obtained at the previous shooting position are displayed side by side. Next, by designating the recognition button on the operation control screen, the sign recognition process is started, and the sign recognition process for a certain number of captured image data is executed from the designated position. As a result, the icon of the shooting position on the map on the main display screen is displayed again by changing the color and size of the icon according to the recognition evaluation value. The change basis of the icon display uses the total value of the first evaluation value or the second evaluation value described up to the previous stage.

  When it takes a long time to obtain the second evaluation value, it may be performed using only the high-speed first evaluation value, and is not limited. However, when only the first evaluation value is used, there is a possibility that the appearance rate of label candidates described later will be redundant. In addition, this recognition processing may be performed in batches for a large number of scenes before starting the operator operation. To edit the operator, the recognition result can be obtained by simply calling the file, and the recognition process can be shortened.

  Next, the operator searches the icon display on the main display screen for a highly evaluated place indicating the possibility of the appearance of this sign, and designates its position with the pointing device, thereby obtaining a captured image of the scene where the road sign appears. Search for. At the same time, a photographed image of the scene position immediately before the scene position is also retrieved and displayed side by side on the sub display screen section. Next, in order to confirm whether or not the sign candidate obtained by the sign recognition process has a correct result, the scene image image of the sub display screen is displayed by pressing the screen switching button in the sub display screen section (230). And switch between the main display screen map and icon display screen. A display example after the switching is shown in FIG. As a result, the main display screen becomes captured image data, the image position can be designated in detail, and the marker candidate region can be erroneously extracted or the region where extraction has failed can be designated. The circumscribing rectangle of the sign image as a recognition candidate is displayed on the main screen by the sign recognition process, and when an arbitrary position within the circumscribing rectangle is designated, the identified sign candidate icon is displayed on the candidate display screen. To be. If the recognition result is incorrect, it can be corrected so as to be a correct sign by operating the icon selection menu on the candidate display screen with the pointing device. On the other hand, if the selected scene-captured image contains a sign but is not extracted as a candidate for a sign, the pointing device designates an outline rectangle containing the sign and determines the image by the operator. And correct sign data is input by operating the icon selection menu on the candidate display screen.

  A series of operations from confirmation to correction of the recognition result is performed using a scene captured image displayed on the main display screen (220), and the specific operation content will be described with reference to FIG. . When recognition processing is executed by operator specification, not only the result position is displayed as a circumscribed rectangle on the main display screen (220), but also the identification result is indicated by a predetermined position on the recognition sequence display screen (260). (For example, in the scene shown in FIG. 10, the recognition result is displayed side by side in the third matrix of the recognition sequence display screen). Furthermore, when the inside of the circumscribed rectangle indicating the recognition result is designated on the main display screen by the pointing device, the image inside the circumscribed rectangle is cut out and enlarged on the candidate display screen (240) and identified in descending order of recognition evaluation value. Display results side by side with icons. The operator looks at such a display result and determines the identification result of the sign. If the determination is correct, the recognition result is displayed as an icon on the candidate display screen. On the other hand, when the position of the recognition result is extracted by mistake, the circumscribed rectangle is deleted and deleted from the list of recognition results. On the other hand, when the sign to be recognized is not extracted, the circumscribed rectangle is manually designated by the pointing device, the image at the designated position is displayed on the candidate display screen, and the icon that becomes the corresponding sign is determined by the operator. Select it and add it to the list of recognition results.

In such a series of operations, only the position on the screen of the sign appearing in each scene and the identification data are input. As the sign database, it is necessary to input the actual position (latitude / longitude and height) of the sign. As shown in many well-known examples, this marker position survey was recognized as a camera parameter (focal length, installation angle, etc.) if the size and height of the marker to be recognized were assumed to be fixed values. It can be obtained from the target pixel size by simple coordinate conversion calculation (see Patent Document 1 and Patent Document 2). However, since the sign size and installation location are not fixed values in practice, a radar distance meter, a laser length measuring device, or a triangulation method using stereo vision using two cameras is used. In the case of the present embodiment, since continuous imaging image scenes by the measurement vehicle are obtained, calculation can be performed by triangulation using a plurality of imaging image scenes arranged in the traveling direction by one camera. This method can be realized by the calculation method disclosed in Patent Document 2, but the surveying calculation method is not limited thereto. Rather, what is a problem in surveying calculation using a plurality of captured image scene sequences is a method for selecting an imaging scene of the same sign that requires at least two. However, each scene contains a large error in the position information due to poor GPS reception status, and includes disturbances such as the sign image hidden behind the traveling vehicle and not becoming a regular image size. There are things to do. Therefore, in this embodiment, the positioning data based on the combination with the least error is adopted from the calculated positioning values calculated from the plurality of combinations by using the characteristic that the marker to be positioned appears in a plurality of scenes. This method is used. A specific calculation method will be described using the positioning calculation model of FIG. This calculation model actually requires a three-dimensional model description, but will be described with reference to a drawing disassembled into a horizontal projection and a vertical projection. As a precondition, the following values are given for each image.
B: Camera latitude
L: Camera longitude
Z: Elevation of the camera
Ah: Camera axis direction angle to true north direction
Av: Camera axial direction angle with respect to horizontal plane
Ar: Camera rotation angle around the camera axis
Px: Sign image coordinates
Py: Sign image coordinates
F: Camera focal length
W x H: Number of pixels
μ: Resolution
Avo: Camera installation vertical angle The observation accuracy of the camera position and azimuth is as follows.
ΔX: Camera horizontal position accuracy ΔZ: Camera vertical position accuracy Δα: Camera azimuth angle accuracy ΔP: Sign coordinate accuracy on the image The sign coordinates are calculated from the camera position (B, L, Z) of each image and from the camera to the sign Using the azimuth angle, the coordinates of the position that intersects this can be obtained. However, errors are included in the coordinates and azimuth, and this value cannot be obtained by simple mathematical processing. It is possible to calculate the marker coordinates based on the error minimization calculation using the position and azimuth for each image data where the marker is photographed, but the accuracy of the marker coordinates in the observation system Is assumed to be in the order of meters, so the calculation can be simplified as follows: It is based on the calculation of sign coordinates by the forward crossing method on the horizontal and vertical planes using two pieces of image data, and it is the most accurate among the results calculated for multiple sets of images taken with the same sign. The policy is to adopt a calculation result that is estimated to be high. The side lengths S1 and S2 of the triangle formed by the distance L between the two points, the average azimuth angle β, and the intersection angle 2θ are obtained as in the following equation 2.

以下に、前述の前方交会法を適応して、標識の座標（B,L,Z）を算定する方法を示す。
カメラ中心から標識への方位角の計算：
画像座標（Px,Py）から、画像中心を原点とした、水平・鉛直座標（Xp、Yp）の計算水平面に対して、カメラ水平軸がAr回転しているものとすると、下記数３で算定できる。

The method for calculating the coordinates (B, L, Z) of the sign by applying the forward crossing method described above is shown below.
Calculation of the azimuth from the camera center to the sign:
From the image coordinates (Px, Py), with the center of the image as the origin, the horizontal and vertical coordinates (Xp, Yp) are calculated. it can.

カメラ中心から標識への方位角の計算：
カメラの観測方位角と、上の画像座標と焦点距離Fとの間の角度の和として

次に基線の計算方法について述べる。まず、水平面計算における基線は2点の経緯度より、基線の長さL、方位角αは、下記数４。

Calculation of the azimuth from the camera center to the sign:
As the sum of the observation azimuth of the camera and the angle between the image coordinates above and the focal length F

Next, the calculation method of the baseline is described. First, the base line in the horizontal plane calculation is based on the longitude and latitude of two points, the length L of the base line, and the azimuth angle α is the following formula 4.

次に鉛直面計算における基線は、2点の標高差を用いて、基線の長さL、方位角αを算定する。下記数５。

Next, for the baseline in the vertical plane calculation, the length L of the baseline and the azimuth angle α are calculated using the difference in elevation between the two points. Number 5 below.

2点のカメラ中心から標識への方位角を、それぞれαｈ１、αｈ２とすると、水平面前方交会法計算に必要な基本数は、下記数６として求められる。

Assuming that the azimuth angles from the two camera centers to the sign are αh1 and αh2, respectively, the basic number required for the horizontal plane forward intersection calculation is obtained as

同様に、鉛直面の前方交会法計算に必要な基本数は下記数７となる。

Similarly, the basic number required for the forward crossing calculation on the vertical plane is the following formula 7.

前方交会法により求められた、三角形の辺長S1を用いて、以下の通り標識座標を算定する。下記数８。

Using the side length S1 of the triangle obtained by the forward intersection method, the marker coordinates are calculated as follows. Number 8 below.

以上より、標識座標は、下記数９として得られる。

From the above, the marker coordinates are obtained as Equation 9 below.

次に既に述べた測位対象とする標識が複数のシーンに出現する標識の認識特性について、図12を用いて更に詳しく説明を加える。図12(a)で示すように、計測車両のカメラの位置がiの位置にある場合のカメラ撮影有効範囲をP_iS-P_iEとすると、計測車両位置の移動（１の位置から５の位置まで移動したとする）に伴いカメラ撮影有効範囲は図12(b)で示すようにP_1S-P_5Eに広がる。この場合、道路に設置された標識に該当する部分の画像の大きさが変わるため、標識候補としての評価値も変化する。その具体的な評価値としては既に図5の126示される標識領域として評価する第１の評価値、又は図6の133で示されるような標識識別のための第２の評価値、及びそれらの積算値等を用いる。そこでこれらの認識評価値を反映するための一方法として、図12(b)右側には各カメラ位置において、円の大きさを変えた表示様式を示している。即ち、標識候補としての評価値を最小値の１から最大値100にいたる幅に正規化したとして、その値に応じた半径の円表示を行なうことにより、どの場所で最も標識候補が観測できているのかが容易に分かるようにする。図12(b)の右側には、その評価値を円の大きさで反映した例を示しており、最も遠く離れた１の位置から標識画像の大きさに比例して２の位置及び３の位置と評価値が大きくなり、４の位置においては標識画像の一部が画面から隠れることの影響により認識評価値が下がりはじめ、５の位置においては完全にカメラ視野から外れて評価値がなくなるような例を示している。更にこの標識の設置位置は、図12(c)で示されるように、複数の標識が接近して設置される場合があり、その場合の標識候補の評価値は一部のカメラ位置では重なることになる。例えば図12(c)右側の標識候補の評価値を示す円表示において、１の位置から４の位置に至る範囲では標識Ａの画像に対する評価値が円の大きさに反映されるが、４の位置から８の位置に至る範囲では、標識Ｂの画像に対する評価値が円の大きさで反映され表示される。この場合４の位置においては、標識Ａと標識Ｂの両方の評価値が加算されることになるため、個々の標識単体の画像に対する評価値よりも大きく表示される。

Next, the recognition characteristics of a marker in which the marker to be positioned already described appears in a plurality of scenes will be described in more detail with reference to FIG. As shown in FIG. 12 (a), if the camera capture effective range when the camera of the measurement vehicle is at position i is P _iS -P _iE , the measurement vehicle position is moved (positions 1 to 5). As shown in FIG. 12 (b), the effective camera shooting range extends to P _1S -P _5E . In this case, since the size of the image corresponding to the sign installed on the road changes, the evaluation value as the sign candidate also changes. As the specific evaluation value, the first evaluation value that is already evaluated as the labeled region 126 shown in FIG. 5, or the second evaluation value for identifying the label as shown by 133 in FIG. An integrated value or the like is used. Therefore, as a method for reflecting these recognition evaluation values, the right side of FIG. 12 (b) shows a display format in which the size of the circle is changed at each camera position. That is, assuming that the evaluation value as a marker candidate is normalized to a range from the minimum value 1 to the maximum value 100, by displaying a circle with a radius according to the value, the marker candidate can be observed most in any place. Make it easy to see if you are. The right side of FIG. 12 (b) shows an example in which the evaluation value is reflected in the size of a circle. From the position 1 that is farthest away, the positions 2 and 3 are proportional to the size of the sign image. The position and evaluation value increase, and at position 4, the recognition evaluation value begins to drop due to the influence of part of the sign image hidden from the screen. At position 5, the evaluation value disappears completely from the camera field of view. An example is shown. Furthermore, as shown in Fig. 12 (c), there are cases where a plurality of signs are placed close to each other, and the evaluation value of the sign candidate in that case overlaps at some camera positions. become. For example, in the circle display indicating the evaluation value of the label candidate on the right side of FIG. 12 (c), the evaluation value for the image of the label A is reflected in the size of the circle in the range from the position 1 to the position 4. In the range from the position to the position 8, the evaluation value for the image of the sign B is reflected and displayed in the size of the circle. In this case, since the evaluation values of both the signs A and B are added at the position 4, they are displayed larger than the evaluation values for the images of the individual signs alone.

  The operator who recognizes and edits the sign data designates the largest display position among the circles of evaluation values of the sign candidate displayed on the map of the main display screen after the recognition process is performed by specifying the recognition menu. In addition to confirming and correcting the marker recognition, survey calculation of the marker position is executed from the evaluation positions of the series of recognition candidates. Then, the position coordinates obtained as a result of the survey calculation are displayed as icons together with the identification information of the signs at the corresponding positions on the main display screen. The process of operation from this series of survey calculations to icon display on the map will be described with reference to FIG. On the map displayed on the main display screen, the evaluation values of candidate markers are displayed in circles with different radii, and a series of results specified by the operator are arranged on the recognition sequence display screen (260) at the bottom of the screen. Is displayed. Among them, the operator selects and designates the recognition result corresponding to the series (290) having a high evaluation value from the recognition series display screen (the portion 300 indicated by the dotted line in the lower part of FIG. 13), and within that range. The position of the included sign is the target of the survey calculation. As a part to be subjected to the survey calculation, two or more sign images are selected. When two or more mark images are selected, the survey calculation is repeated by the number of combinations of two image scenes. Among them, the one determined to have the least uncertain error component is adopted as the representative position of the sign. Then, an icon corresponding to the identified sign type is displayed at a position on the map corresponding to the survey calculation position (301), so that the operator can confirm the validity of the survey calculation. When the display position of the sign icon deviates greatly, the operator can correct it to an appropriate position. As a designation method for displaying the sign candidate on the recognition sequence display screen, the position of the point indicating the camera position on the main display screen is designated from the screen using the pointing device 2, and the camera position within a certain range is determined therefrom. A series is selected and displayed on the recognition series display screen.

  In the second embodiment, it is assumed that the second embodiment is implemented with the same hardware configuration as the first embodiment. Focusing on the interactive recognition editing unit (12) in the central processing unit having this configuration, the configuration of the screen interface unit for the processing will be described. In the first embodiment, after the recognition process is executed, the first presentation to the operator is a circle indicating the map display on the main display screen as shown in FIG. 8 and the evaluation values of the marker candidates displayed on the map. The operator designates the part with the high evaluation value from the pointing device and displays it on the recognition sequence display screen, and performs the survey calculation for the specified sequence set from among them, on the main screen. It was a flow to confirm the total amount position. The reason for adopting such an interface step is that if the recognition rate of the sign is not high, the sign recognition error will affect the survey calculation, so the sign recognition error will be corrected before the survey calculation and recognized as much as possible. It aims to prevent the influence of errors on survey calculations. On the other hand, when the sign recognition rate is high, it is not always necessary to separate such recognition / correction process and survey calculation process. That is, a survey calculation using a plurality of label candidates is continuously executed from the recognition of the sign, and the map of the main display screen using the identified sign and position information (latitude / longitude / height). Show icons on top. Therefore, the interface first presented to the operator is the one displayed on the map at the position corresponding to the survey calculation using the icon of the recognized marker type as shown in FIG. In order to confirm the recognition result, the operator designates a portion having a high evaluation value of the marker candidate on the map as in the first embodiment, and displays the recognition result side by side on the recognition sequence display screen (260). Or confirming a sign icon displayed on the map and tracing a corresponding sign candidate correspondingly. This recognition processing and survey calculation processing are executed together to reduce the man-hours involved in confirmation editing by the operator.

  By continuing the recognition / editing process up to the position where the above procedure is repeatedly specified, the operator can efficiently construct the sign information database. As in this embodiment, according to the present invention, it is possible to construct a database at a lower cost than a database construction cost by a known marker data input method.

1 shows an example of a system configuration for carrying out the present invention. The example of the flow of the whole process by this invention. The example of the color recognition and noise removal process flow in a present Example. An example of a bandwidth threshold for color recognition. The example of the flow of a candidate area | region extraction process. An example of the flow of a label identification process. An example of a color and shape combination dictionary. 6 illustrates an example of a configuration of an interactive recognition editing screen according to the first embodiment. An example of the display screen replaced with the sub display screen in Example 1. FIG. 6 shows an example of confirmation edit screen operation in the first embodiment. FIG. 4 is an example of a positioning calculation model in the first embodiment. An example of the measurement vehicle camera position and effective imaging | photography range in Example 1. FIG. An example of the interactive recognition edit screen after survey calculation in Example 1. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Central processing unit part which comprises the system of this invention 2 Instruction device part which comprises the system of this invention 3 Display unit which comprises the system of this invention
210 Example of operation control screen of interactive recognition edit screen that makes up the first embodiment
220 Example of the main display screen of the interactive recognition editing screen according to the first embodiment
230 Sub-display screen example of interactive recognition editing screen that forms the first embodiment
240 Candidate Display Screen Example of Interactive Recognition Editing Screen that Forms Example 1
250 Example of the position information display screen of the interactive recognition editing screen according to the first embodiment
260 A recognition sequence display screen example of the interactive recognition editing screen according to the first embodiment.

Claims (11)

  It has a sign recognition unit that receives road images and an editing unit that performs interactive editing from screen operations. The recognition unit is activated from the editing unit, and the recognition result is displayed on the screen together with the recognition evaluation value. A road sign data input system that performs confirmation and editing operations from the part with a high evaluation value.   In addition to the configuration shown in claim 1, a surveying unit that calculates a position from a sign image is provided, and a survey calculation is performed using a sign image of a part that has been edited based on a recognition evaluation value. Road sign data input system.   The recognition evaluation value according to claim 1, wherein a first evaluation value for evaluating the extraction accuracy of a candidate area where a sign is present, and a second evaluation value for evaluating recognition for identifying a sign from a candidate area image after extraction A road sign data input system comprising:   4. The road sign data input system according to claim 1, wherein only the first evaluation value of claim 2 is used in the recognition evaluation value of claim 1.   A road characterized in that the recognition result and the recognition evaluation value of claim 1 are both displayed on the screen as characters and symbols having a size, color and shape proportional to the recognized evaluation value. Indicator data input system.   The editing unit according to claim 1, comprising a main screen for displaying a map and a road image in a wide range and a sub-screen for displaying them in a small size, and displaying the main screen content and the sub-screen content interchangeably with each other. An input system for road sign data.   In the editing unit according to claim 6, when a shooting position of a road image is displayed on a map with a symbol or a character on a shooting camera trajectory, and the position is selected by an instruction device, a plurality of shooting positions adjacent to the position are selected. A road sign data input system that displays a road image on a sub-screen.   8. The road sign data input system according to claim 7, wherein a plurality of the sub-screens are displayed in an adjacent order in the method for displaying the road image on the sub-screen.   In the survey calculation method according to claim 2, when the survey calculation is performed using the sign image of the portion where the editing operation based on the recognition evaluation value is performed, the calculation is performed using at least two consecutive multiple sign images, A road sign data input system using a calculated value with the least error among surveying calculations of a combination of two or more sheets.   In the survey calculation using the sign image of claim 2, using the survey calculation value, it is possible to display using the icon of the recognized sign to the corresponding position on the map and to move and edit the position. An input system for road sign data.   2. The road sign data according to claim 1, wherein the recognition results are displayed side by side in the order obtained by the recognition unit, and the corresponding positions on the map and road image are displayed as marks such as circles or circumscribed rectangles. Input system.

Images