JP4978615B2 - Target identification device - Google Patents

Description

  The present invention relates to an object specifying device that measures the position of a road sign using, for example, a three-dimensional point cloud model generated based on data measured by a laser radar and a camera image.

In recent years, products combining GIS (Geographical Information System) and GPS (Global Positioning System) represented by car navigation systems and the like have been widely used. Development of ITS (Intelligent Transport Systems) is being promoted in order to eliminate traffic accidents and traffic congestion. In addition, each local government records information on features (road signs, kilometer posts, traffic lights, white lines, guardrails, utility poles, etc.) around roads (roads, roads, roadsides, etc.) as road management ledgers. Advancement and high accuracy of the ledger are desired.
For these reasons, it is important to easily acquire position information of features around the road with high accuracy.

However, it is very difficult to easily acquire the position information of the features around the road with high accuracy.
For example, the creation of a road management ledger that records the position information of features around roads at a scale of 1/500 requires high-precision surveying, so stationary using GPS and a total station that measures distance and angle Surveying is being performed. On the national road, there may be approximately 2000 features to be measured in a 30 km round-trip section. Therefore, enormous costs and time are required for the advancement and accuracy of road management ledgers nationwide.

Therefore, MMS (Mobile Mapping System) that acquires various types of information from vehicles traveling on the road has been attracting attention and researched and developed for the purpose of reducing information collection time and cost.
Patent Document 1 discloses an example of an apparatus for surveying road alignment.
JP-A-10-267650 JP 2007-078409 A JP 2005-025497 A Mitsunobu Yoshida, 4 others, “Mobile Mapping System”, Mitsubishi Electric Technical Report, Mitsubishi Electric Engineering Co., Ltd., August 25, 2007, Vol. 81, No. 8, p. 15-18

  An object of the present invention is to make it possible to easily acquire highly accurate position information of features around a road using, for example, MMS.

  The object specifying device of the present invention is different from a point cloud data storage unit that stores, using a storage device, three-dimensional point cloud data that represents a feature in three dimensions using a plurality of three-dimensional point data indicating three-dimensional coordinates. An image data storage unit that stores first image data and second image data representing a captured two-dimensional image using a storage device, and the first image data stored in the image data storage unit Target imaging that uses a CPU (Central Processing Unit) to specify a target imaging range in which a specific target is captured from a two-dimensional image and a two-dimensional image represented by the second image data stored in the image data storage unit The CPU is used for the two-dimensional image represented by the first image data stored in the image data storage unit and the range specifying unit and the three-dimensional point group data stored in the point group data storage unit. A first point cloud image projecting unit that projects and a three-dimensional point cloud projected by the first point cloud image projecting unit on the target imaging range of the first image data specified by the target imaging range specifying unit A first target projection point group extraction unit that extracts data using a CPU, and the three-dimensional point group data extracted by the first target projection point group extraction unit based on the three-dimensional coordinates indicated by the three-dimensional point data. A second group image stored in the image data storage unit, and a plurality of group point group data grouped by the point group grouping unit. A second point cloud image projecting unit that projects using a CPU on a two-dimensional image represented by the data, and a second target image capturing range specified by the target capturing range specifying unit; Point cloud image projection unit A second target projection point group extraction unit that extracts each projected group point group data using a CPU, and each group point group data grouped by the point group grouping unit includes the second target projection point group extraction unit. Is compared with the group point cloud data extracted by the point cloud grouping unit, and the group point cloud data having the largest residual rate of the three-dimensional point data among the group point cloud data grouped by the point group grouping unit is determined as the specific object. A target point group specifying unit that uses a CPU as target point group data expressed in three dimensions is provided.

  According to the present invention, for example, target point cloud data representing a feature to be measured can be specified using three-dimensional point cloud data and image data. And the position of the measurement target feature can be calculated with high accuracy based on the target point cloud data. Further, the three-dimensional point cloud data and the image data can be easily obtained by traveling on the road with MMS. That is, according to the present invention, it is possible to easily acquire highly accurate position information of features around the road.

Embodiment 1 FIG.
FIG. 1 is a functional configuration diagram of a road sign position measurement system 900 according to the first embodiment.
As shown in FIG. 1, a road sign position measurement system 900 according to Embodiment 1 is a measurement vehicle 200 (MMS) that travels on a road and acquires image data around the road and distance direction data representing features around the road. ), And a road sign position measuring device 100 that measures the three-dimensional coordinates of the road sign based on the image data and the distance direction data acquired by the measurement vehicle 200.
In the description of the first embodiment, the three-dimensional coordinates of the road sign are measured. However, the road sign position measuring apparatus 100 calculates the three-dimensional coordinates of other features around the road (white lines on the road surface, power poles, traffic lights, traveling vehicles, etc.). It may be measured in the same way as a road sign.

The measuring vehicle 200 (MMS) is fixedly installed on a top plate 201 (base) provided with a GPS receiver 210, a three-axis gyro 220, an odometer 230, a camera 240, and an LRF 250 (Laser Range Finder) on a roof portion. ing.
The measurement vehicle 200 includes an observation data storage unit 291, an image data storage unit 292, and a distance azimuth that store data acquired by the GPS receiver 210, the three-axis gyro 220, the odometer 230, the camera 240, and the LRF 250 using a storage device. A point cloud storage unit 293 is provided.

  Three GPS receivers 210 are provided in the measurement vehicle 200, and each GPS receiver 210 observes a carrier wave of a positioning signal transmitted from each of a plurality of GPS satellites. Then, each GPS receiver 210, based on the observation information of the carrier wave, a single positioning result indicating the pseudo distance between each GPS satellite and the own GPS receiver 210, the antenna position of the own GPS receiver 210 in three-dimensional coordinates, the carrier wave And the navigation message (satellite orbit information) indicated by the positioning signal are stored in the observation data storage unit 291 as observation data.

  The triaxial gyro 220 measures the angular acceleration (also called rate) of the azimuth angle [yaw angle], elevation angle [pitch angle], and rotation angle [roll angle] of the measurement vehicle 200, and the angular acceleration at each time is used as angular acceleration data. Store in the observation data storage unit 291.

  The odometer 230 measures the traveling speed of the measuring vehicle 200 and stores the traveling speed at each time in the observation data storage unit 291 as vehicle speed data.

The camera 240 images the front (traveling direction) of the traveling measuring vehicle 200 at different times, and stores the image data of each captured image in the image data storage unit 292. That is, the camera 240 stores road image data captured at different points in the image data storage unit 292.
Hereinafter, the image data measurement vehicle 200 is in the storage unit 292 the image data of the road by the camera 240 is captured when passing through the P ₀ point (hereinafter referred to as captured image A) and the measurement vehicle 200 has passed the P ₁ point Assume that road image data (hereinafter referred to as captured image B) captured by the camera 240 is stored. Further, it is assumed that the same road sign is reflected in the captured image A and the captured image B. In the first embodiment, the road sign position measuring apparatus 100 measures the three-dimensional coordinates of the road signs shown in the captured image A and the captured image B.

The LRF 250 irradiates the side of the traveling measuring vehicle 200 with a laser, observes the laser reflected back to the feature, and locates the feature based on the observation values at each observation point where the laser was observed. This is a laser radar that generates a plurality of point data (point group data) indicating a relative distance and a relative direction with respect to the LRF 250. The LRF 250 stores the generated point cloud data in the distance bearing point cloud storage unit 293 as distance bearing point cloud data representing the feature.
Hereinafter, it is assumed that the distance azimuth point cloud data stored in the distance azimuth point cloud storage unit 293 includes point cloud data representing road signs reflected in the captured image A and the captured image B. In the first embodiment, the road sign position measuring apparatus 100 measures three-dimensional coordinates for the road sign represented by the distance direction point group data.

The road sign position measuring device 100 (target identification device) includes a vehicle position / orientation locating unit 110, a road sign image recognizing unit 120, a three-dimensional point cloud model restoring unit 130, a point cloud image projecting unit 140, a grouping unit 150, a sign point group. The identification unit 160 and the marker position calculation unit 170 are provided.
Further, the road sign position measuring apparatus 100 includes a vehicle position / posture storage unit 191, a recognition image storage unit 192, a three-dimensional point cloud model storage unit 193, and a two-dimensional projection point cloud storage unit 194 that store each data using a storage device. And a marker candidate point cloud storage unit 195. Further, the road sign position measuring apparatus 100 inputs each data stored from the observation data storage unit 291, the image data storage unit 292, and the distance azimuth point group storage unit 293 of the measurement vehicle 200, and stores each input data as a storage device. Use and remember. That is, the road sign position measuring apparatus 100 includes a storage unit (not shown) corresponding to the observation data storage unit 291, the image data storage unit 292, and the distance / azimuth point group storage unit 293.

The vehicle position / orientation locating unit 110 is based on the observation data of the GPS receiver 210 stored in the observation data storage unit 291, the angular acceleration data of the three-axis gyro 220, and the vehicle speed data of the odometer 230. Coordinates and attitude angles (roll, pitch, yaw) are standardized.
The vehicle position / orientation locating unit 110 stores the three-dimensional coordinates and the attitude angle of the measurement vehicle 200 in the vehicle position / orientation storage unit 191 as vehicle position / orientation data.

The road sign image recognition unit 120 (target imaging range specifying unit) includes a two-dimensional image and a captured image B (second image data) represented by the captured image A (first image data) stored in the image data storage unit 292. A sign imaging range Aa / Ba (target imaging range, window) in which a road sign (specific object) is imaged from the represented two-dimensional image is specified using a CPU (Central Processing Unit).
Road sign image recognition unit 120 stores labeled imaging range _{A a,} recognition image _{A r} indicating the labeled imaging range _{B a,} the recognition image _{B r} to the recognition image storage unit 192.

The three-dimensional point cloud model restoration unit 130 three-dimensionally identifies features around the road by using a plurality of three-dimensional point data indicating three-dimensional coordinates based on the distance direction point cloud data stored in the distance direction point group storage unit 293. 3D point cloud data (hereinafter referred to as a 3D point cloud model) is generated.
The 3D point cloud model restoring unit 130 stores the 3D point cloud model in the 3D point cloud model storage unit 193 (point cloud data storage unit).

The point cloud image projecting unit 140 (first point cloud image projecting unit) projects the 3D point cloud model onto the 2D image represented by the captured image A using the CPU.
Hereinafter, the point cloud data projected on the two-dimensional image represented by the captured image A is referred to as an image projection point cloud A _ip .
Further, the point cloud image projecting unit 140 (second point cloud image projecting unit) has the group point cloud data (hereinafter referred to as marker candidate point cloud A _c) identified by the marker point cloud label limiting unit 153 from the image projected point cloud A _ip. Are projected onto the two-dimensional image represented by the captured image B using the CPU.
Hereinafter, the point cloud data projected on the two-dimensional image represented by the captured image B is referred to as an image projection point cloud B _ip .
The point group image projection unit 140 stores the image projection point group A _ip and the image projection point group B _ip in the two-dimensional projection point group storage unit 194.

The grouping unit 150 includes a marker projection point group extraction unit 151, a marker point group labeling unit 152, and a marker point group label limiting unit 153, and includes point group data projected on the marker imaging range A _a (hereinafter, marker projection point group A _sp). And the sign projection point group A _sp is grouped into group point groups representing different features, and a group point group (sign candidate point group A _c ) having the same size as a road sign is identified. To do.
Further, the grouping unit 150 labeled imaging range B _a the projected point cloud data (hereinafter referred to as labeled projected point cloud B _sp) for identifying the labeled candidate point group B _c.
The grouping unit 150 stores the label candidate point group A _c and the label candidate point group B _c in the label candidate point group storage unit 195.

Labeled projected point cloud extracting section 151 (first target projection point cloud extracting section) extracts the projected image projected point cloud A _ip signs imaging range A _a using CPU.
Moreover, labeled projected point cloud extracting section 151 (second target projection point cloud extracting section) extracts an image projected point cloud B _ip projected on labeled imaging range B _a with CPU.
Hereinafter, the point cloud data projected onto the marker imaging range A _a is referred to as a marker projection point cloud A _sp , and the point cloud data projected onto the marker imaging range B _a is referred to as a marker projection point group B _sp .

The marker point group labeling unit 152 (point group grouping unit) groups the marker projection point group _Asp as a plurality of group point group data using the CPU based on the three-dimensional coordinates indicated by the three-dimensional point data.
In particular, the sign point group labeling unit 152 sets a three-dimensional space (hereinafter referred to as voxel space) divided into a plurality of three-dimensional regions (hereinafter referred to as voxels), and sets the sign projection point group _Asp to each three-dimensional area. Voting is performed on the voxel space based on the three-dimensional coordinates indicated by the point data, and each three-dimensional point data voted for each adjacent voxel is set as one group point group data.

The sign point cloud label limiting unit 153 (point cloud group limiting unit) uses the CPU to determine the size of the feature (group body) represented by each of the plurality of group point cloud data based on the 3D coordinates indicated by the 3D point data. The size of the identified feature is compared with the size of the road sign using the CPU, and each group point cloud data (marker candidate point group A _c ) corresponding to the size of the feature corresponds to the size of the road sign. Use to specify.

The marker point group specifying unit 160 (target point group specifying unit) compares each group point cloud data constituting the marker candidate point group _Ac with the group point cloud data constituting the marker projection point group _Bsp , and the marker candidate point labeled three-dimensional point group representing the largest group point group data is three-dimensional point data of the residual rate in the labeled projected point cloud B _sp in each group point cloud data constituting the group a _c road signs in three dimensions (subject The point cloud data is specified using a CPU.
The sign point group specifying unit 160 stores the sign three-dimensional point group in the sign three-dimensional point group storage unit 196.

  The sign position calculation unit 170 (target position calculation unit) uses the CPU as the position of the road sign using the three-dimensional coordinates indicating the center of the road sign based on the three-dimensional coordinates indicated by the three-dimensional point data constituting the sign three-dimensional point group. Use to calculate.

FIG. 2 is a diagram illustrating an example of hardware resources of the road sign position measurement apparatus 100 according to the first embodiment.
FIG. 3 is a diagram illustrating an example of hardware resources of the measurement vehicle 200 according to the first embodiment.
2 and 3, the road sign position measurement device 100 and the measurement vehicle 200 are also referred to as a CPU 911 (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor for executing a program. ). The CPU 911 is connected to the ROM 913, the RAM 914, the communication board 915, and the magnetic disk device 920 via the bus 912, and controls these hardware devices. Instead of the magnetic disk device 920, another storage device (for example, a semiconductor memory such as a RAM or a flash memory) may be used.
Further, the road sign position measuring device 100 includes a display device 901 having a CRT (Cathode / Ray / Tube) or LCD (liquid crystal) display screen, a keyboard 902 (Key / Board: K / B), a mouse 903, an FDD 904 (Flexible Disk / Drive) and CDD905 (compact disk device), and the CPU 911 also controls these hardware devices.
The measurement vehicle 200 further includes a camera 240, a GPS receiver 210, a three-axis gyro 220, an odometer 230, and an LRF 250. The CPU 911 also controls these hardware devices.

The RAM 914 is an example of a volatile memory. The storage medium of the ROM 913 and the magnetic disk device 920 is an example of a nonvolatile memory. These are examples of a storage device, a storage device, or a storage unit. A storage device in which input data is stored is an example of an input device, an input device, or an input unit, and a storage device in which output data is stored is an example of an output device, an output device, or an output unit.
The communication board 915, the keyboard 902, the mouse 903, the FDD 904, and the CDD 905 are examples of input devices, input devices, or input units.
The communication board 915, the display device 901, the FDD 904, and the CDD 905 are examples of an output device, an output device, or an output unit.

  The communication board 915 is wired or wirelessly connected to a communication network such as a LAN (Local Area Network), the Internet, a WAN (Wide Area Network) such as ISDN, and a telephone line.

  The magnetic disk device 920 stores an OS 921 (operating system), a program group 923, and a file group 924. The programs in the program group 923 are executed by the CPU 911 and the OS 921.

  The program group 923 stores a program for executing a function described as “˜unit” in the embodiment. The program is read and executed by the CPU 911.

In the file group 924, in the embodiment, result data such as “determination result”, “calculation result of”, “processing result of” when executing the function of “to part”, “to part” The data to be passed between programs that execute the function “,” other information, data, signal values, variable values, and parameters are stored as items “˜file” and “˜database”. Data such as captured images, observation data, distance azimuth points, vehicle positions / postures, recognition images, 3D point cloud models, image projection points, sign candidate points, sign 3D points, and road sign positions are file groups 924 is an example of what is included in 924.
The “˜file” and “˜database” are stored in a recording medium such as a disk or a memory. Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. Used for CPU operations such as calculation, processing, output, printing, and display. Information, data, signal values, variable values, and parameters are temporarily stored in the main memory, cache memory, and buffer memory during the CPU operations of extraction, search, reference, comparison, operation, calculation, processing, output, printing, and display. Is remembered.
In addition, arrows in the flowcharts described in the embodiments mainly indicate input / output of data and signals, and the data and signal values are recorded in the memory of the RAM 914, the magnetic disk of the magnetic disk device 920, and other recording media. . Data and signal values are transmitted online via a bus 912, signal lines, cables, or other transmission media.

  In addition, what is described as “˜unit” in the embodiment may be “˜circuit”, “˜device”, “˜device”, and “˜step”, “˜procedure”, “˜”. Processing ". That is, what is described as “˜unit” may be realized by firmware stored in the ROM 913. Alternatively, it may be implemented only by software, or only by hardware such as elements, devices, substrates, and wirings, by a combination of software and hardware, or by a combination of firmware. Firmware and software are stored as programs on a magnetic disk or other recording medium. The program is read by the CPU 911 and executed by the CPU 911. That is, the target specifying program causes the computer to function as “to part”. Alternatively, the procedure or method of “to part” is executed by a computer.

FIG. 4 is an arrangement relation diagram of the measurement vehicle 200 in the first embodiment.
FIG. 5 is a diagram illustrating a laser measurement method of the LRF 250 in the first embodiment.
A laser measurement method for LRF 250 in the first embodiment will be described with reference to FIGS.
As shown in FIG. 4, the measurement vehicle 200 is installed with a top plate 201 fixed on the roof of the vehicle body 202, and the top plate 201 has three GPS receivers 210 (210 a to 210 b), a camera 240 and three axes. The gyro 220 and the LRF 250 are fixedly installed.
The coordinates of the installation point O of the 3-axis gyro 220 and the attitude angle at the installation point of the 3-axis gyro 220 are defined as the position and the attitude angle of the measurement vehicle 200.

Hereinafter, the polar coordinate system xyz having the installation point O of the three-axis gyro 220 as the origin O (0, 0, 0) is defined as a vehicle coordinate system. The coordinates and attitude angles represented in the vehicle coordinate system indicate relative coordinates and relative attitude angles with respect to the measurement vehicle 200.
The z axis indicates the traveling direction (longitudinal direction, depth direction, vertical direction) of the measuring vehicle 200, the x axis indicates the width direction (lateral direction) of the measuring vehicle 200, and the y axis indicates the height direction (vertical) of the measuring vehicle 200. Direction), an angle Φ around the z axis indicates a rotation angle (roll angle), an angle θ around the x axis indicates an elevation angle (pitch angle), and an angle ψ around the y axis indicates an azimuth angle (yaw angle). Show.
In addition, the coordinates and posture angles of the camera 240 in the vehicle coordinate system are (Δx _cam , Δy _cam , Δz _cam ), (Φ _cam , θ _cam , ψ _cam ). Hereinafter, (Δx _cam , Δy _cam , Δz _cam ) and (Φ _cam , θ _cam , ψ _cam ) are referred to as camera mounting offsets.
In addition, the coordinates and posture angles of the LRF 250 in the vehicle coordinate system are (Δx _lrf , Δy _lrf , Δz _lrf ), (Φ _lrf , θ _lrf , ψ _lrf ). Hereinafter, (Δx _lrf , Δy _lrf , Δz _lrf ) and (Φ _lrf , θ _lrf , ψ _lrf ) are referred to as LRF attachment offsets.
Similarly, the coordinates and attitude angles of each GPS receiver 210 in the vehicle coordinate system are set as a receiver mounting offset.
A polar coordinate system representing the relative coordinates and relative attitude angles with respect to the camera 240 with the position of the camera 240 as the origin is taken as the camera coordinate system, and a polar coordinate system representing the relative coordinates and relative attitude angles with respect to the LRF 250 with the position of the LRF 250 as the origin. And a polar coordinate system representing a relative coordinate and a relative attitude angle with respect to each GPS receiver 210 with the position of each GPS receiver 210 as an origin is defined as a receiver coordinate system.

  While traveling, the measuring vehicle 200 continuously observes a carrier wave from a GPS satellite by each GPS receiver 210, images the surroundings of the road ahead by the camera 240, and performs laser measurement on the features on the left side by the LRF 250.

In FIG. 5, the LRF 250 irradiates the laser while swinging the nose 180 degrees in the vertical direction (y-axis direction) in units of 1 °, and observes the laser reflected on the feature to measure the distance and direction to the feature. And repeat this.
For example, the LRF 250 acquires distance azimuth point group data indicating the distance to the feature and the azimuth (direction) for each point of the scanning line S _a at the point a, and the scanning line S at the point b where the measurement vehicle 200 travels. Distance azimuth point cloud data is acquired for each point _b . Then, the LRF 250 repeats this laser measurement while the measuring vehicle 200 is traveling, thereby acquiring distance azimuth point data for each feature located on the left side of the road on which the measuring vehicle 200 has traveled. In addition, the measurement vehicle 200 travels back and forth on the road and acquires distance azimuth point cloud data for each feature located on the left and right sides of the road. When the LRF 250 performs laser measurement, the measurement vehicle 200 does not stop at points a, b,..., But the LRF 250 continuously performs laser measurement while the measurement vehicle 200 continuously travels.
For example, the LRF 250 performs laser irradiation and observation while moving the nose up and down at such a speed that the interval between the points indicated by the distance and azimuth point group data is about 10 to 14 cm. Alternatively, the traveling speed of the measurement vehicle 200 is adjusted.

FIG. 6 is a flowchart showing the road sign position measurement process in the first embodiment.
FIG. 7 is a flowchart showing pre-processing (S100) in the first embodiment.
A road sign position measurement process (target identification method) executed by the road sign position measurement apparatus 100 according to the first embodiment will be described below with reference to FIGS. 6 and 7.
Each part which comprises the road sign position measuring apparatus 100 performs each process demonstrated below using CPU911.

<S100: Pre-processing>
First, recognizing a recognition image A _r and recognition image B _r specified range that is reflected road sign with respect to a road marker position measurement device 100 measures the vehicle 200 captured images captured at different points of A and the captured image B The image is stored in the image storage unit 192.
In addition, the road sign position measuring apparatus 100 stores a three-dimensional point group model generated based on the distance direction point group data measured by the measurement vehicle 200 in the three-dimensional point group model storage unit 193.
Here, details of the preprocessing (S100) will be described with reference to FIG.

<S110>
First, the vehicle position and orientation determination unit 110 determines the position and orientation angle of the measurement vehicle 200 at each time using a specific coordinate system. For example, the vehicle position and orientation determination unit 110 determines the position and orientation angle of the measurement vehicle 200 in a latitude / longitude coordinate system in which coordinates are represented by latitude, longitude, and altitude. Further, for example, the vehicle position / orientation locating unit 110 uses the travel start point of the measurement vehicle 200 as an origin and the measurement vehicle 200 in an ENU (East North Up) coordinate system in which coordinates are expressed in the east direction, the north direction, and the upward direction (vertical direction). The position and attitude angle of

At this time, the vehicle position and orientation determination unit 110 inputs observation data, angular acceleration data, and vehicle speed data at each time from the observation data storage unit 291, and the position and posture angle of the measurement vehicle 200 at each time based on the input data. (Roll angle, pitch angle, yaw angle) are standardized. The vehicle position and orientation determination unit 110 stores the position and posture angle of the measurement vehicle 200 at each determined time in the vehicle position and orientation storage unit 191 as vehicle position and orientation data.
Here, the observation data includes a single positioning result, pseudorange, carrier phase, satellite orbit information, and the like.

For example, the vehicle position / orientation locating unit 110 corrects the single positioning result of the GPS receiver 210 by the receiver mounting offset, and determines the three-dimensional coordinates of the measurement vehicle 200.
Further, for example, the vehicle position / orientation locating unit 110 adds the change amount obtained by integrating the angular acceleration data of the three-axis gyro 220 and the vehicle speed data of the odometer 230 to the previous orientation value, thereby performing the three-dimensional measurement of the measurement vehicle 200. Position coordinates and attitude angles.
Further, for example, the vehicle position / orientation locating unit 110 estimates an error included in the result of dead reckoning by the Kalman filter process, and corrects the result of dead reckoning using the estimation error to determine the three-dimensional coordinates and the attitude angle of the measurement vehicle 200. To do. The Kalman filter process is a process for estimating an error of a state quantity based on both equations of a state equation that models the dynamics of the state quantity and an observation equation that formulates the relationship between the observed quantity and the state quantity. In the Kalman filter processing, the vehicle position / orientation locating unit 110 uses the observation residual of the pseudorange and the observation residual of the carrier wave phase as the observation amount, and the localization result by dead reckoning as the state amount. For example, the vehicle position / orientation locating unit 110 calculates the distance between the GPS receiver 210 and the GPS satellite based on the position of the GPS receiver 210 calculated by dead reckoning and the position of the GPS satellite indicated by the satellite orbit information. The difference from the pseudorange indicated by the observation data is taken as the observation residual. Further, for example, the vehicle position / orientation locating unit 110 uses the position of the GPS receiver 210 calculated by dead reckoning and the position of the GPS satellite indicated by the satellite orbit information to indicate the LOS (LOS) indicating the direction of the GPS satellite as viewed from the GPS receiver 210. (Line Of Light) vector, a baseline vector indicating the distance and direction between the GPS receivers 210 based on the receiver mounting offset, a carrier phase calculated based on the LOS vector and the baseline vector, and calculated The difference between the carrier phase and the carrier phase indicated by the observation data is defined as an observation residual.

<S120: Target imaging range specifying process>
Next, the road sign image recognizing unit 120 specifies the sign image pickup range A _a and the sign image pickup range B _{a in} which the same road sign is captured from the captured image A and the captured image B.
At this time, the road sign image recognition unit 120 inputs the captured image A and the captured image B from the image data storage unit 292, and performs arbitrary image recognition processing for recognizing the same road sign for the captured image A and the captured image B. Do. The road sign image recognition unit 120, based on the image recognition processing result, to identify the labeled imaging range A _a in which the road sign is captured on the captured image A, is the road sign on the captured image B identifying a is reflected labeled imaging range B _a. For example, the road sign image recognition unit 120 identifies the sign image pickup range from the picked-up image by pattern matching with the feature information (shape, color, pattern, etc.) of each road sign stored in advance in the storage device. Alternatively, the road sign image recognition unit 120 inputs a plurality of captured images from the image data storage unit 292, performs image recognition processing on each captured image, and captures two captured images that have been able to identify the sign imaging range. A and picked-up image B are selected.
Further, the road sign image recognition unit 120 acquires the position and posture angle of the measurement vehicle 200 when the captured image A is captured and the position and posture angle of the measurement vehicle 200 when the captured image B is captured from the vehicle position and orientation storage unit 191. To do.
Next, the road sign image recognition unit 120 uses the position and orientation angle of the measurement vehicle 200 and the camera mounting offset to capture the position and orientation angle of the camera 240 when capturing the captured image A and the captured image B when capturing the captured image B. The position and posture angle of the camera 240 are calculated in a specific coordinate system. Assume that the camera mounting offset is stored in advance in the storage device.
The road sign image recognition unit 120 recognizes the captured image A, the sign imaging range A _a, and the recognition image _Ar indicating the position / posture angle of the camera 240 at the time of capturing the captured image A, the captured image B, and the sign imaging range B _a. In addition, the recognition image storage unit 192 stores the recognition image _Br indicating the position / posture angle of the camera 240 at the time of capturing the captured image B.
Hereinafter, the sign imaging range is represented by a quadrangle and indicates two coordinate values on a diagonal line representing the quadrangle in an image coordinate system in which the upper left of the captured image is the origin and the pixel of the captured image is a unit. . For example, in this image coordinate system, when the captured image is 1600 pixels wide × 1200 pixels long, the coordinates at the upper left of the image are (0, 0), the coordinates at the upper right of the image are (1599, 0), and the coordinates at the lower left of the image are ( 0, 1199), the coordinates on the lower right of the image are (1599, 1199).

<S130>
Then, the three-dimensional point cloud model restoration unit 130 generates a three-dimensional point cloud model that represents the features around the road in three dimensions using a plurality of three-dimensional point data indicating the three-dimensional coordinates.
At this time, the three-dimensional point cloud model restoration unit 130 inputs distance azimuth point cloud data from the distance azimuth point cloud storage unit 293.
Next, the three-dimensional point cloud model restoration unit 130 uses the relative distance and relative azimuth (direction) with respect to the LRF 250 indicated by each point of the distance azimuth point cloud data for each point of the distance azimuth point cloud data in the LRF coordinate system. Calculate three-dimensional coordinates.
Next, the three-dimensional point cloud model restoration unit 130 acquires the position and posture angle of the measurement vehicle 200 at each time from the vehicle position / posture storage unit 191.
Then, the three-dimensional point cloud model restoration unit 130 performs three-dimensional affine transformation on the three-dimensional coordinates of each point in the LRF coordinate system based on the position and posture angle of the measurement vehicle 200 and the LRF attachment offset, thereby obtaining a three-dimensional point cloud. Generate a model. It is assumed that the LRF attachment offset is stored in the storage device in advance.
Each point of the three-dimensional point cloud model is represented by a specific coordinate system (for example, the ENU coordinate system and the vehicle coordinate system).
The 3D point cloud model restoration unit 130 stores the generated 3D point cloud model in the 3D point cloud model storage unit 193.

  Here, the three-dimensional affine transformation will be described. Each symbol used in the following equations corresponds to the symbol shown in FIG.

First, the three-dimensional coordinates (x ₀ , y _0, z ₀ ) of each point in the LRF coordinate system obtained from the distance azimuth point group data are relative positions (Δx _lrf , Δy _lrf , Δz _lrf ) of the LRF 250 with respect to the measurement vehicle 200. ) And relative posture (Φ _lrf , θ _lrf , ψ _lrf ) (LRF mounting offset), and converted into the three-dimensional coordinates (x ₂ , y ₂ , z ₂ ) of the vehicle coordinate system by the following expressions 1 and 2 Is done.

The three-dimensional coordinates (x ₂ , y ₂ , z ₂ ) of each point in the vehicle coordinate system are the coordinates (Ev, Nv, Uv) and the attitude angle (Φv, θv, ψv) of the measurement vehicle 200 in the ENU coordinate system. _Are converted into three-dimensional coordinates (E _lrf , N _lrf , U _lrf ) of the ENU coordinate system by the following formulas 3 and 4 using.

FIG. 8 is a diagram illustrating an example of a three-dimensional point group model generated by the three-dimensional point group model restoration unit 130 according to the first embodiment.
The three-dimensional point cloud model generated by the three-dimensional point cloud model restoration unit 130 is displayed as a three-dimensional image as shown in FIG. FIG. 8 shows that road signs, utility poles, and trees are arranged in order on the side of the road in parallel with the roadway (traveling direction of the measuring vehicle 200).

  It is assumed that the 3D point cloud model restoration unit 130 generates a 3D point cloud model only for points whose horizontal distance from the LRF 250 is a predetermined value or less. For example, FIG. 8 shows that a feature that exists beyond the reed is not included in the three-dimensional point cloud model.

Next, returning to FIG. 6, the road sign position measurement processing after the preprocessing (S100) will be described.
The road sign position measuring apparatus 100 according to the first embodiment recognizes a road recognized on a captured image in order to identify which point group represents a road sign among a plurality of points constituting the three-dimensional point cloud model. Identify the point cloud corresponding to the sign.
Therefore, after the preprocessing (S100), the road sign position measuring apparatus 100 projects each point constituting the three-dimensional point cloud model onto the captured image A and the captured image B (S140, S160), and the road among the projected points. Each point located in the marker imaging range A _a and the marker imaging range B _a in which the marker is reflected is extracted (S151, S171), the extracted points are grouped (S152), and the marker imaging range of each group point group A group point group common to A _a and the sign imaging range B _a is specified as a group point group representing a road sign (S180). Then, the road sign position measuring apparatus 100 calculates the three-dimensional coordinates of the road sign based on the three-dimensional coordinates indicated by the points constituting the group point group representing the specified road sign (S190).

FIG. 9 is a conceptual diagram of road sign position measurement processing in the first embodiment.
Here, a two-dimensional captured image A (image plane I ₀ ) obtained by capturing a three-dimensional point cloud model in which road signs, trees, and utility poles are arranged in parallel with the roadway (traveling direction of the measuring vehicle 200) is captured at the point P _0. when projected, road signs located LOS of camera 240 (viewing direction), each group point group representing each tree and the utility pole is projected superimposed on labeling imaging range a _a. However, when projected to the same three-dimensional point cloud model a two-dimensional image captured by the points P ₁ B (image plane I _1), the LOS is the point P ₀ from the point P ₁ to the same road sign since different from the LOS, only groupings that represent a portion of the point cloud groups and trees representing the road signs for labeling imaging range B _a is projected. For this reason, the road sign position measuring apparatus 100 can specify the group point group projected in common to the sign imaging range A _a and the sign imaging range B _a having different viewpoints as a group point group representing a road sign.

  Details of each process of S140 to S190 will be described below with reference to FIG.

<S140: Point Cloud Image Projection Processing A>
First, the point cloud image projection unit 140 projects a three-dimensional point cloud model onto the captured image A.
At this time, the point cloud image projection unit 140 acquires the recognition image A from the recognition image storage unit 192 and acquires the 3D point cloud model from the 3D point cloud model storage unit 193. Here, as described above, the recognition image A indicates the position / posture angle of the camera 240 when the captured image A, the sign capturing range _Aa, and the captured image A are captured.
Then, the points constituting the 3D point cloud model are projected onto the captured image A as follows, and the 3D coordinates indicated by the 3D point cloud model and the 2D coordinates after projection onto the captured image A are projected for each point. _Is stored in the two-dimensional projection point group storage unit 194. Hereinafter, an image after projecting the three-dimensional point cloud model is referred to as a projected image. Further, the two-dimensional coordinates of the image projection point group indicate coordinate values with the upper left corner of the captured image as the origin and the pixel of the captured image as a unit, as in the above-described marker imaging range.

FIG. 10 is a conceptual diagram showing projection of a point of three-dimensional coordinates on the two-dimensional image plane I in the first embodiment.
Each symbol used in the following equations corresponds to the symbol shown in FIGS.

First, the coordinates (x _lrf , y _lrf , z _lrf ) of the vehicle coordinate system indicated by the point D of the three-dimensional point cloud model are the coordinates (Δx _cam , Δy _cam , Δz _cam ) and attitude angle of the camera 240 in the vehicle coordinate system. The coordinates are converted into the coordinates (x _cam , y _cam , z _cam ) of the camera coordinate system by the following formulas 5 and 6 using (Φ _cam , θ _cam , ψ _cam ).

A straight line L connecting the coordinates (x _cam , y _cam , z _cam ) of the point D in the camera coordinate system and the camera center C (x _cam0 , y _cam0 , z _cam0 ) is expressed by the following formulas 7, 8, and 9: And is represented by Equation 10. The camera center C indicates the center of the image sensor of the camera 240 or the origin of the camera coordinates.

Further, assuming that the camera 240 is an ideal pinhole camera, the image plane I is expressed by the following expression 11 using the focal length f. The focal length f indicates the distance between the camera center C and the center C ′ of the image plane I.
z = f (Formula 11)

  The point cloud image projecting unit 140 calculates the coordinates of the intersection D ′ between the image plane I and the straight line L using the above formulas 7 to 11. The coordinates of the intersection D ′ are the two-dimensional coordinates of the point D of the three-dimensional point cloud model projected on the captured image.

FIG. 11 is a diagram showing an example of the projection image A in the first embodiment.
For example, point group image projection unit 140 generates a projection image A _p as shown in FIG. 11 by projecting the three-dimensional point cloud model to the captured image A. In FIG. 11, the points constituting the image projection point group A _{ip on} which the three-dimensional point group model is projected correspond to each feature in the captured image A by the above point group image projection processing A (S140). It shows that. For example, in FIG. 11, the right image obtained by enlarging the α portion of the left image indicates that the point group is projected along the extension direction of the utility pole.

<S151: Target Projection Point Group Extraction Processing A>
6, is labeled projected point cloud extracting section 151 extracts the labeled imaging range A _a a projected labeled projected point cloud A _sp the captured image A.
At this time, labeled projected point cloud extracting section 151 obtains a recognition image A _r from the recognition image storage unit 192, obtains an image projected point cloud A _ip from a two-dimensional projection point group storage unit 194. Here, the position / orientation angle of the camera 240 at the time of capturing the captured image A, the sign capturing range _Aa, and the captured image A are shown, and the image projection point group A _ip represents the three-dimensional coordinates and the two on the captured image A for each point. Dimensional coordinates are shown.
Then, the sign projection point group extraction unit 151 extracts a point group located in the sign imaging range A _a from the image projection point group A _ip based on the sign imaging range A _a and the two-dimensional coordinates on the captured image A. Then, the extracted point group is set as a marker projection point group _Asp . Labeling projected point cloud A _sp showing the two-dimensional coordinates on the three-dimensional coordinates and the captured image A for each point located within the labeled imaging range A _a.

FIG. 12 is a diagram illustrating an example of the marker projection point group A _sp [two-dimensional] according to the first embodiment.
FIG. 13 is a diagram illustrating an example of the marker projection point group A _sp [three-dimensional] according to the first embodiment.
For example, labeled projected point cloud extracting section 151, as shown in FIG. 12, it extracts a recognition image A _r represents labeled imaging range A _a label projected point cloud projected on the A _sp. In Figure 12, the left side shows the overall recognition image A _r, shows a labeled imaging range A _a which is extended to the center. In addition, as shown in the right figure, the sign imaging range Aa of FIG. 12 includes a sign for _a pedestrian crossing, a utility pole and a tree located behind the sign.
That is, as shown in FIG. 13, the sign projection point group _Asp includes a point group representing a road sign, a point group representing a utility pole, and a point group representing a tree.

<S152: Point cloud grouping process>
In FIG. 6, the marker point group labeling unit 152 labels the marker projection point group _Asp for each group.
For example, with respect to the sign projection point group _Asp shown in FIG. 13, the sign point group labeling unit 152 performs a point group representing a road sign, a point group representing a power pole, and a point group representing a tree as follows. And set a label for each group point cloud.
FIG. 14 is a diagram illustrating a voxel model according to the first embodiment.
FIG. 15 is a diagram illustrating labeling (grouping) in the first embodiment.
Details of the point group grouping process (S152) will be described below with reference to FIGS.

First, the sign point cloud labeling unit 152 sets a voxel space (S152a [see the left diagram in FIG. 14)].
The voxel space is obtained by dividing a three-dimensional space into a cubic lattice having a cube (voxel) as a unit. For example, the sign point group labeling unit 152 searches for the minimum value of each of the coordinate components x, y, and z with respect to the three-dimensional coordinates of each point constituting the sign projection point group _Asp , and one side is set with the coordinate indicated by the minimum value as the origin. A voxel space is set with a 20 centimeter voxel as a division unit. However, the marker point cloud labeling unit 152 may divide the three-dimensional space by other solid shapes such as a rectangular parallelepiped or a triangular pyramid.
Next, the sign point group labeling unit 152 votes each point constituting the sign projection point group _Asp to the corresponding voxel based on the three-dimensional coordinates. At this time, the sign point group labeling unit 152 performs the sign projection located in each voxel based on the three-dimensional coordinates of each point constituting the sign projection point group _Asp and the coordinates of the three-dimensional space formed by each voxel. The point of the point group A _sp is specified. Hereinafter, identifies voxels each of the points composing the labeled projected point group A _sp is located, that vote be associated with a point and voxel labeled projected point cloud A _sp (S152b [14 middle panel Reference).
Then, the landmark group labeling unit 152 extracts one voxel found the point labeled projected point cloud A _sp as a voxel model (S152c [14 right view Reference).
Next, the marker point cloud labeling unit 152 groups the adjacent voxels with respect to the voxel model as one group, and sets labels for the group point cloud voted for each voxel constituting the group for each group. (S152d [see the right figure of FIG. 15]).

  For example, the sign point cloud labeling unit 152 performs grouping by connecting adjacent voxels in the surrounding 26 directions, and sets a label for each group. That is, a certain voxel belongs to the same group as other voxels located in a 3 × 3 × 3 cube centered on itself.

  FIG. 15 shows that the voxel models are grouped into four groups, group 1, group 2, group 3, and group 4. The group point cloud voted for the voxel of group 1 is identified by label 1, the group point cloud voted for the voxel of group 2 is identified by label 2, and the group point cloud voted for the voxel of group 3 is labeled 3 And the group point group voted for the voxel of group 4 is identified by label 4.

  If the size of the voxel set in S152a is too large, the point cloud representing the road sign and the point cloud representing the other feature are voted for the same or adjacent voxels, and the point cloud representing the road sign and the other Cannot be grouped into point clouds that represent features. Also, if the size of the voxel is too small, each point representing the same road sign is voted for a voxel that is distant without voting to the same or adjacent voxel, and each point representing the road sign is Cannot be a group point cloud. Therefore, the sign point cloud labeling unit 152 sets the voxel size according to the interval between the points measured by the LRF 250, the size of the road sign to be measured, and the like. For example, the marker point cloud labeling unit 152 sets a voxel with a length of one side that is 1.5 times or more and 2.0 times or less of an interval (for example, 10 to 14 centimeters) between points measured by the LRF 250. Further, for example, the sign point cloud labeling unit 152 sets the voxel to be about 0.1 to 0.5 times the length of one side (for example, 40 to 180 centimeters) of the road sign to be measured. Set.

FIG. 16 is a diagram illustrating an example of the marker projection point group A _sp [three-dimensional] after the point group grouping process (S152) in the first embodiment.
The sign point group labeling unit 152 performs the point group grouping process (S152) to identify the mark projection point group A _sp [three-dimensional] shown in FIG. They can be grouped into point groups identified by label 3.

<S153: Point cloud group limited processing>
6, is labeled point group label limited portion 153 is labeled point group labeling unit 152 to limit the group set point at which the labeled candidate point group A _c for each group point group that has been grouped.
At this time, the sign point group label limiting unit 153 calculates the size (width, height, depth) of the feature represented by the group point group based on the three-dimensional coordinates of each point for each group point group.
Next, the sign point cloud label limiting unit 153 compares the size range indicating the size of the road sign to be measured with the size of the feature represented by each group point cloud, and the size of the feature is the size range of the road sign. The group point group which is inside is specified. It is assumed that the size range of the road sign is defined in the storage device in advance.
Then, each labeled point group label limited section 153 point group constituting each group point group identified and stored as a label candidate point group A _c signs the candidate point group storage unit 195, a point other than the labeled candidate point group A _c Remove the group. The sign candidate point group A indicates the three-dimensional coordinates of each point and the two-dimensional coordinates on the captured image A of each point with respect to each group point group representing a feature having the same size as the road sign to be measured.

FIG. 17 is a graph illustrating an example of a range in which the marker projection point group A _sp [three-dimensional] in the first embodiment exists on the East-North plane.
FIG. 18 is a graph showing an example of a range in which the marker projection point group A _sp [3D] in the first embodiment exists on the East-Up plane.
FIG. 19 is a diagram illustrating an example of the label candidate point group A _c [three-dimensional] according to the first embodiment.
Each point constituting each group point group identified by label 1, label 2 or label 3 shown in FIG. 16 exists in the East-North plane as shown in FIG. 17, and in the East-Up plane, as shown in FIG. Exists. FIG. 17 shows the existence range of each point when viewed from above, and FIG. 18 shows the existence range of each point when viewed from the side. The scales in FIGS. 17 and 18 indicate the distance from the point where the LRF 250 starts laser measurement. In FIG. 17, the LOS (line-of-sight direction) of the camera 240 is the direction from the upper right of the graph, and in FIG. 18, the LOS of the camera 240 is the direction from the lower right of the graph.
Here, the size range of a road sign (for example, a pedestrian crossing sign) to be measured is set to 0.4 to 1.8 meters in the width direction and 0.4 to 1.8 meters in the height direction. In FIG. 18, each group point group identified by label 1, label 2 or label 3 satisfies the size condition in the height direction. On the other hand, in FIG. 17, each group point group identified by label 1 or label 3 satisfies the size condition in the width direction, but the group point group identified by label 2 does not satisfy the size condition in the width direction. .
In this case, the sign point cloud label limiting unit 153 deletes the group point cloud identified by “label 2”, and the label name of the group point cloud identified by “label 3” is changed to “label 3” as shown in FIG. Update to label 2 ". In FIG. 19, the group point group identified by “label 1” and the group point group identified by “label 2 (old label 1)” are the marker candidate point group _Ac .

<S160: Point Cloud Image Projection Processing B>
In FIG. 6, the point cloud image projecting unit 140 projects the marker candidate point cloud _Ac onto the captured image B.
At this time, point cloud image projection section 140 acquires the recognition image B from the recognition image storage unit 192, acquires the labeled candidate point group A _c from the labeled candidate point group storage unit 195.
The point cloud image projection unit 140, like the point cloud image projection processing A (S140), by projecting the labeled candidate point group A _c in the captured image B, the image projected point cloud B _ip projected in the captured image B This is stored in the two-dimensional projection point group storage unit 194.
However, point cloud image projection section 140 is not to project the all points constituting the three-dimensional point cloud model to the captured image B, and constitute the labeled candidate point group A _c the label point group label limited section 153 is limited Only the points are projected onto the captured image B.
Here, the recognition image _Br indicates the position / posture angle of the camera 240 when the captured image B, the sign capturing range _Ba, and the captured image B are captured. Moreover, labeled candidate point group A _c represents the two-dimensional coordinates on the three-dimensional coordinates and the captured image A of each point for each point cloud group representing the feature is the size of the road sign and comparable measurement object image projected point cloud B _ip denotes the two-dimensional coordinates on the three-dimensional coordinates and the captured image B of each point labeled candidate point group a _c.

<S171: Target Projection Point Group Extraction Processing B>
Next, the labeled projected point cloud extracting section 151 extracts the labeled imaging range B _a the projected labeled projected point cloud B _sp as labeled candidate point group B _c the captured image B.
At this time, it labeled projected point cloud extracting section 151 obtains a recognition image B _r from the recognition image storage unit 192, obtains an image projected point cloud B _ip from a two-dimensional projection point group storage unit 194.
Then, labeled projected point cloud extracting section 151, like the target projected point cloud extracting process A (S151), it extracts the projected into labeled imaging range B _a label projected point cloud B _sp. Labeling projected point cloud extracting section 151 stores the extracted labeled projected point cloud B _sp as labeled candidate point group B _c signs the candidate point group storage unit 195.
That is, the labeled projected point cloud extracting section 151 is a labeled candidate point group B _c narrow down to the projected point of the plurality of points constituting the labeled candidate point group A _c signs imaging range B _a.

<S180: Target Point Cloud Specifying Process>
Next, the sign point group specifying unit 160 compares the sign candidate point group _Ac and the sign candidate point group B _c to specify a sign three-dimensional point group representing a road sign.
At this time, labeled point group specification unit 160 acquires the labeled candidate point group A _c and labeled candidate point group B _c from the labeled candidate point group storage unit 195.
The landmarks group specifier 160 compares the respective groups point group constituting each group point group and labeled candidate point group B _c constituting the labeled candidate point group A _c, the largest group matching score of the points is The point cloud is stored in the sign three-dimensional point cloud storage unit 196 as a sign three-dimensional point cloud representing a road sign. Each point constituting the labeled candidate point group B _c is a point one constituting the labeled candidate point group A _c, handled as belonging to the same group as the group of labeled candidate point group A _c. Landmarks group specifier 160, in each group point group constituting the labeled candidate point group A _c, the proportion of the points constituting the group point group remained as points constituting the labeled candidate point group B _c The highest group point cloud is defined as a marker 3D point cloud. The sign three-dimensional point group indicates the three-dimensional coordinates of each point representing the road sign.

Here, as shown in FIG. 9, the captured image A captured at the point P ₀ and the captured image B captured at the point P ₁ differ in the LOS (line-of-sight direction) of the camera 240 with respect to the road sign. It does not coincide with the imaging range a labeled projected point cloud projected on _a a _sp and labeled imaging range B _a the projected labeled projected point cloud B _sp. However, since both the captured image A and the captured image B represent the same road sign, the group point groups representing the road signs in the sign projection point group A and the sign projection point group B coincide. That is, when the marker candidate point group A _c and the marker candidate point group B _c shown in FIG. 19 are compared, the group point group representing the road sign (label 2) is the group point group representing the tree (label 1). The degree of coincidence / residual rate of the point group becomes higher. For this reason, as shown in FIG. 20, the sign point group specifying unit 160 rejects the group point group (label 1) representing the tree, and sets the group point group (label 2) representing the road sign as the sign three-dimensional point group. Can be identified.
FIG. 20 is a diagram showing an example of a marker three-dimensional point group in the first embodiment.

<S190: Target position calculation process>
In FIG. 6, the sign position calculation unit 170 calculates the position of the road sign based on the sign three-dimensional point group.
At this time, the marker position calculation unit 170 acquires the marker three-dimensional point group from the marker three-dimensional point group storage unit 196.
Then, the sign position calculation unit 170 calculates the three-dimensional coordinates of the center point (center of gravity) of the road sign as the position of the road sign based on the three-dimensional coordinates of each point indicated by the sign three-dimensional point group. At this time, the sign position calculation unit 170 calculates the average value of the coordinate components x, y, and z as the position of the road sign with respect to the three-dimensional coordinates of each point constituting the sign three-dimensional point group. For example, for each of the coordinate components x, y, and z, the marker position calculation unit 170 searches for the minimum value and the maximum value from the coordinate values of each point, and the value obtained by dividing the total value of the minimum value and the maximum value by 2 Is calculated as the position of the road sign. Further, for example, the sign position calculation unit 170 calculates the value obtained by summing the coordinate values of each point for each of the coordinate components x, y, and z and dividing the total value by the number of points as the position of the road sign.

FIG. 21 is a diagram showing a road sign position in the first embodiment.
As shown in FIG. 21, the sign position calculation unit 170 calculates the center point of the sign three-dimensional point group shown in FIG. 20 as the road sign position. FIG. 21 shows a sign three-dimensional point group (left figure) representing a road sign from the front and a sign three-dimensional point group (right figure) representing a road sign from the side.

  In the first embodiment, a road sign that measures a road sign position using a three-dimensional restoration point cloud (three-dimensional point cloud model) calculated from data obtained by LRF based on a highly accurate vehicle body position and orientation and a camera image. The position measurement method has been described. The feature of this method is that the parallax between multiple camera images with different viewpoints is used, so that the laser measurement points corresponding to the features other than the signs hidden around or behind the sign are rejected and reflected directly from the sign. The position of the marker is measured using only the returning laser point.

In the road sign position measuring apparatus 100 according to the first embodiment, the point cloud image projection unit 140 projects the marker candidate point cloud Ac on the captured image B instead of the 3D point cloud model (S160), and the point data to be processed is processed. The number can be reduced and the processing time can be shortened.
Further, the road sign position measuring device 100 limits the group point cloud based on the size of the feature represented by each group point cloud by the sign point cloud label limiting unit 153 (S153), so that the road sign size is large. It is possible to prevent erroneous detection of group point clouds of different features as mark three-dimensional point clouds.

Embodiment 2. FIG.
In the second embodiment, a road sign position measurement process that does not execute the point cloud group limitation process (S153) will be described.
Hereinafter, items different from the first embodiment will be mainly described, and items that will not be described are the same as those in the first embodiment.

FIG. 22 is a flowchart showing a road sign position measurement process in the second embodiment.
As shown in FIG. 22, the road sign position measuring apparatus 100 according to the second embodiment does not execute the point cloud group limiting process (S153).
Then, the point cloud grouping process (S152b), the landmark group labeling unit 152 stores each group point group constituting the labeled projected point group A _sp as labeled candidate point group A _c signs the candidate point group storage unit 195, in point cloud image projection processing B (S160b), the point cloud image projection section 140 projects the labeled candidate point group a _c the label point group labeling unit 152 stores the captured image B.

  As a result, the road sign position measuring apparatus 100 can calculate the position of the road sign even without the sign point cloud label limiting unit 153.

Embodiment 3 FIG.
In the third embodiment, a road sign position measurement process for executing a target point group identification process (S180) different from that in the first embodiment will be described.
Hereinafter, items different from the first embodiment will be mainly described, and items that will not be described are the same as those in the first embodiment.

FIG. 23 is a flowchart showing a road sign position measurement process in the third embodiment.
The road sign position measurement process in the third embodiment will be described below with reference to FIG.
Comparison in the target point cloud specifying process (S180c), landmarks group specifier 160 matching degree for each group point group calculated by comparing the labeled candidate point group A _c and labeled candidate point group B _c (the residual ratio) Then, one group point group whose degree of coincidence is greater than the degree of coincidence of all other group point groups by a predetermined threshold or more is specified as a marker three-dimensional point group.
In S181c, the road sign position measurement apparatus 100 has the sign point group specifying unit 160 that cannot specify the sign three-dimensional point group, that is, the degree of coincidence of each group point group is not more than a predetermined threshold. In this case, the processing is repeated from the point cloud image projection processing B (S160c).
At this time, the point cloud image projection processing B (S160c), the point cloud image projection section 140 projects a different captured image x the labeled candidate point group A _c new (third captured image).
Thereafter, comparison labeled projected point cloud extracting section 151 and the extracts labeled candidate point group _{x c} (S171c), group landmarks group specifier 160 labeled candidate point _{A c} and labeled candidate point group _{x c} captured image x Thus, the marker three-dimensional point group is specified.
The road sign position measuring apparatus 100 repeats S160c to S181c using different captured images until the sign three-dimensional point group can be specified.

  In other words, the road sign position measuring apparatus 100 is picked up at a point where the picked-up image A and the picked-up image B are close to each other, and even if the difference in viewpoint is small, the other picked-up image having a large viewpoint difference from the picked-up image A Can be used to identify the marked three-dimensional point cloud. Thereby, the road sign position measuring apparatus 100 can prevent erroneous detection of the sign three-dimensional point group.

Embodiment 4 FIG.
In the fourth embodiment, a road sign position measurement process for executing a point group image projection process (S140d) to a point group group limitation process (S153d) for a three-dimensional point group model for each captured image will be described.
Hereinafter, items different from the first embodiment will be mainly described, and items that will not be described are the same as those in the first embodiment.

FIG. 24 is a flowchart showing a road sign position measurement process in the fourth embodiment.
The road sign position measurement process in the fourth embodiment will be described below with reference to FIG.
In the point cloud image projection process (S140d), the point cloud image projection unit 140 projects a three-dimensional point cloud model on the captured image A and the captured image B.
Then, the labeled projected point cloud extracting section 151 extracts labeling projected point cloud _{A sp} and labeled projected point group _{B sp} (S151d), the landmark group labeling unit 152 labels the projected point cloud _{A sp} and labeled projected point group _{B sp} the grouped (S152d), the landmark group label limited section 153 identifies the labeled candidate point group _{a c} and labeled candidate point group _{B c} (S153d).

  Thereby, the road sign position measuring apparatus 100 can execute the point cloud image projection process (S140d) to the point cloud group limitation process (S153d) in parallel for each captured image, and can shorten the processing time.

The road sign position measuring apparatus 100 may omit the point group group limiting process (S153d) as in the second embodiment.
Further, the road sign position measuring apparatus 100 may process three or more captured images, and may identify a sign three-dimensional point group by comparing three or more sign candidate point groups.

1 is a functional configuration diagram of a road sign position measurement system 900 according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of hardware resources of the road sign position measurement apparatus 100 according to the first embodiment. FIG. 3 is a diagram illustrating an example of hardware resources of the measurement vehicle 200 according to the first embodiment. FIG. 3 is an arrangement relation diagram of the measurement vehicle 200 in the first embodiment. FIG. 4 shows a laser measurement method for LRF 250 in the first embodiment. 5 is a flowchart showing a road sign position measurement process in the first embodiment. 5 is a flowchart showing preprocessing (S100) in the first embodiment. FIG. 4 is a diagram illustrating an example of a three-dimensional point cloud model generated by a three-dimensional point cloud model restoration unit according to Embodiment 1. FIG. 3 is a conceptual diagram of road sign position measurement processing in the first embodiment. 3 is a conceptual diagram showing projection of a point of three-dimensional coordinates on a two-dimensional image plane I in Embodiment 1. FIG. FIG. 6 shows an example of a projection image A in the first embodiment. FIG. 6 shows an example of a marker projection point group A [two-dimensional] in the first embodiment. FIG. 6 shows an example of a marker projection point group A [3D] in the first embodiment. FIG. 3 shows a voxel model in the first embodiment. FIG. 3 is a diagram illustrating labeling (grouping) in the first embodiment. The figure which shows an example of the marker projection point group A [three dimensions] after the point group grouping process (S152) in Embodiment 1. FIG. 6 is a graph showing an example of a range in which a marker projection point group A [3D] in the first embodiment exists on an East-North plane. 6 is a graph showing an example of a range in which a marker projection point group A [3D] in the first embodiment exists on an East-Up plane. FIG. 6 is a diagram illustrating an example of a label candidate point group A [3D] according to the first embodiment. FIG. 4 shows an example of a labeled three-dimensional point group in the first embodiment. FIG. 4 shows a road sign position in the first embodiment. 9 is a flowchart showing a road sign position measurement process in the second embodiment. 10 is a flowchart showing a road sign position measurement process in the third embodiment. 9 is a flowchart showing a road sign position measurement process in the fourth embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Road sign position measuring apparatus, 110 Vehicle position and orientation determination part, 120 Road sign image recognition part, 130 Three-dimensional point cloud model reconstruction part, 140 Point cloud image projection part, 150 Grouping part, 151 Sign projection point cloud extraction part, 152 Marking point cloud labeling unit, 153 Marking point cloud label limiting unit, 160 Marking point group specifying unit, 170 Marking point calculation unit, 191 Vehicle position and orientation storage unit, 192 Recognition image storage unit, 193 Three-dimensional point cloud model storage unit, 194 Two-dimensional projection point cloud storage unit, 195 Marking candidate point cloud storage unit, 196 Marking three-dimensional point cloud storage unit, 200 Measuring vehicle, 201 Top plate, 202 Vehicle body, 210 GPS receiver, 220 3-axis gyroscope, 230 odometer, 240 camera, 250 LRF, 291 observation data storage unit, 292 image data storage unit, 293 distance Position group storage unit, 900 road sign position measurement system, 901 display device, 902 keyboard, 903 mouse, 904 FDD, 905 CDD, 911 CPU, 912 bus, 913 ROM, 914 RAM, 915 communication board, 920 magnetic disk device, 921 OS, 923 program group, 924 file group.

Claims (4)

A three-dimensional point group that represents a three-dimensional coordinate and includes a group of three-dimensional points representing a specific object and a group of three-dimensional points representing each of a plurality of features at a first position. A first point cloud image projection unit for projecting onto the captured first image;
A first target imaging range specifying unit that specifies a target imaging range in which a specific target is captured from the first image;
Among the three-dimensional point groups projected onto the first image by the first point group image projecting unit, the three-dimensional point group projected onto the target imaging range identified by the first target imaging range identifying unit. A first target projection point group extraction unit to extract;
A second point group image projection unit for projecting the three-dimensional point group extracted by the first target projection point group extraction unit onto a second image captured at a second position different from the first position; ,
A second target imaging range specifying unit that specifies a target imaging range in which the specific object is captured from the second image;
Of the three-dimensional point group projected onto the second image by the second point group image projection unit, a three-dimensional point group projected onto the target imaging range identified by the second target imaging range identification unit A second target projection point group extraction unit to extract;
The three-dimensional point group extracted by the second target projection point group extraction unit is compared with the three-dimensional point group extracted by the first target projection point group extraction unit, and the first target projection point group is compared. The number of 3D points extracted by the second target projection point group extraction unit with respect to the number of 3D points included in the group of 3D points included in the 3D point group extracted by the extraction unit And a target point group specifying unit for specifying a group of three-dimensional points having the largest ratio as a three-dimensional point group representing the specific target object. A group of 3D point data that represents a feature in three dimensions with a plurality of three-dimensional point data representing three-dimensional coordinates and represents a specific target in three dimensions and a plurality of features respectively. A point cloud data storage unit for storing 3D point cloud data including a group of 3D point data using a storage device;
An image data storage unit that stores first image data and second image data representing a two-dimensional image captured at different positions using a storage device;
A target obtained by capturing a specific object from a two-dimensional image represented by the first image data stored in the image data storage unit and a two-dimensional image represented by the second image data stored in the image data storage unit A target imaging range specifying unit that specifies an imaging range using a CPU (Central Processing Unit);
A first point cloud image that projects the 3D point cloud data stored in the point cloud data storage unit onto a 2D image represented by the first image data stored in the image data storage unit using a CPU. A projection unit;
First target projection for extracting, using a CPU, three-dimensional point group data projected by the first point cloud image projection unit with respect to a target imaging range of first image data identified by the target imaging range identification unit A point cloud extraction unit;
A second projection that uses the CPU to project the 3D point cloud data extracted by the first target projection point cloud extraction unit onto the 2D image represented by the second image data stored in the image data storage unit. A point cloud image projection unit;
Second target projection for extracting, using the CPU, three-dimensional point group data projected by the second point cloud image projection unit with respect to the target imaging range of the second image data identified by the target imaging range identification unit A point cloud extraction unit;
The three-dimensional point group data extracted by the second target projection point group extraction unit and the three-dimensional point group data extracted by the first target projection point group extraction unit are compared, and the first target projection point group is compared. The three-dimensional point data extracted by the second target projection point group extraction unit with respect to the number of three-dimensional points included in the group of the three-dimensional point data included in the three-dimensional point group data extracted by the extraction unit. A target point group specifying unit that uses a CPU to specify a group of three-dimensional point data having the largest number ratio as target point group data representing the specific target in three dimensions. apparatus. A first point group image projecting unit that projects a three-dimensional point group indicating three-dimensional coordinates onto a first image captured at a first position;
A second point group image projecting unit that projects the three-dimensional point group onto a second image captured at a second position different from the first position;
A target imaging range specifying unit that specifies a target imaging range in which a specific target is captured from each of the first image and the second image;
Of the three-dimensional point group projected onto the first image by the first point cloud image projection unit, the three-dimensional image projected onto the target imaging range identified from the first image by the target imaging range identification unit. Of the point group and the three-dimensional point group projected onto the second image by the second point group image projecting unit, projected onto the target imaging range identified from the second image by the target imaging range identifying unit An object specifying apparatus comprising: a target point group specifying unit that compares the three-dimensional point group that has been shared and specifies a common three-dimensional point group as a three-dimensional point group that represents the specific object. A point group data storage unit for storing, using a storage device, three-dimensional point group data representing a feature in three dimensions with a plurality of three-dimensional point data indicating three-dimensional coordinates;
An image data storage unit that stores first image data and second image data representing a two-dimensional image captured at different positions using a storage device;
A target obtained by capturing a specific object from a two-dimensional image represented by the first image data stored in the image data storage unit and a two-dimensional image represented by the second image data stored in the image data storage unit A target imaging range specifying unit that specifies an imaging range using a CPU (Central Processing Unit);
The two-dimensional image represented by the first image data stored in the image data storage unit and the second image data stored in the image data storage unit as the three-dimensional point group data stored in the point group data storage unit A point cloud image projection unit that projects using a CPU to the two-dimensional image represented by
Using the CPU, the 3D point cloud data projected by the point cloud image projection unit with respect to the target imaging range of the first image data specified by the target imaging range specifying unit is extracted as first 3D point group data. And the CPU uses the 3D point cloud data projected by the point cloud image projection unit for the target imaging range of the second image data specified by the target imaging range specifying unit as second 3D point cloud data. A target projection point cloud extraction unit to be extracted using,
The first three-dimensional point group data extracted by the target projection point group extraction unit and the second three-dimensional point group data extracted by the target projection point group extraction unit are compared, and a plurality of common three-dimensional point data And a target point group specifying unit that uses a CPU as target point group data representing the specific target in three dimensions.