JP7163601B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method .

Since paved roads are damaged by the passage of vehicles and the effects of the weather, it is necessary to periodically inspect road properties, and various inspection methods have been proposed. As inspection items for road properties, three indices are defined: the number of cracks, the depth of ruts (irregularities in the width direction of the road), and flatness (irregularities in the direction of travel of the vehicle). For these indicators, cracks are measured visually or using camera images. Also, for rutting and flatness, it is necessary to acquire three-dimensional shape measurement data. Therefore, in general, camera equipment for capturing images of cracks, light-section measurement equipment for measuring rutting, and road surface height sensors installed at two or more locations in front and behind the vehicle for measuring flatness. , and 3 equipment will be used together for the inspection.

In Patent Document 1, the road surface is repeatedly photographed while driving on the road, the photographed original images are joined together to describe the road surface image, and the position and development shape of the crack on the road are specified based on the road surface image. Techniques are disclosed. In Patent Document 1, a reduced road surface image obtained by reducing a road surface image is described, and it is possible to output a crack identification image by superimposing the position and development shape of a crack on the reduced road surface image so as to be identifiable.

In the conventional configuration, while inspections can be performed on more routes, the amount of imaging data and measurement data to be collected is enormous. Therefore, there is a problem that labor is required to extract local damage information.

SUMMARY OF THE INVENTION It is an object of the present invention to make it possible to easily extract local information from a huge amount of imaging data and measurement data.

In order to solve the above-described problems and achieve the object, the present invention provides an acquisition unit that acquires measurement data of road surface conditions in accordance with the vehicle traveling direction of the road surface, and a vehicle traveling direction of the road surface based on the measurement data. a conversion unit that converts to a state characteristic value that indicates the state of the road surface in a predetermined distance unit, a creation unit that creates a report based on the state characteristic value, and a display control unit that displays a screen based on the state characteristic value on the display unit , and the state characteristic value includes indices indicating cracking rate, rutting amount, and flatness, respectively, which are used to calculate the pavement maintenance management index, and the acquisition unit uses the stereo camera to measure the road surface Based on a plurality of stereo camera captured images captured at different times using the is based on three-dimensional point cloud information obtained by integrating a plurality of three-dimensional point cloud information in a common coordinate system and images captured by a stereo camera, with respect to the traveling direction of the vehicle on the road surface. A first state characteristic value, which is a characteristic value, and a second state characteristic value, which is a state characteristic value corresponding to a second distance unit shorter than the first distance unit, are converted.

ADVANTAGE OF THE INVENTION According to this invention, there exists an effect that the measurement regarding three indices in a road property inspection can be performed by simple structure.

FIG. 1 is a diagram for explaining the measurement of the amount of rutting. FIG. 2 is a diagram for explaining flatness measurement. FIG. 3 is a diagram showing an example of an existing record. FIG. 4 is a diagram showing an example of an existing record. FIG. 5 is a diagram illustrating a configuration example of an imaging system according to the embodiment; FIG. 6 is a diagram for explaining an imaging range in the direction of travel of a vehicle by a stereo camera, which is applicable to the embodiment. FIG. 7 is a diagram for explaining an imaging range of a vehicle in the road width direction by a stereo camera, which is applicable to the embodiment. FIG. 8 is a diagram illustrating overlapping of stereo imaging ranges of stereo cameras in the traveling direction of the vehicle, which is applicable to the embodiment. FIG. 9 is a diagram illustrating an example of an imaging system with three stereo cameras, according to an embodiment. FIG. 10 is a block diagram showing an example of a schematic configuration of an imaging system according to the embodiment; FIG. 11 is a block diagram illustrating an example of the hardware configuration of the imaging system according to the embodiment; FIG. 12 is a block diagram showing an example configuration of a stereo camera according to the embodiment. FIG. 13 is a block diagram illustrating an exemplary configuration of an information processing apparatus applicable to the embodiment; FIG. 14 is an exemplary functional block diagram for explaining functions of an information processing apparatus applicable to the embodiment; FIG. 15 is an exemplary flowchart illustrating flatness calculation processing according to the embodiment. FIG. 16 is an exemplary flowchart illustrating depth map generation processing applicable to the embodiment. FIG. 17 is a diagram for explaining trigonometry applicable to the embodiment; FIG. 18 is an example flowchart illustrating camera position and orientation estimation processing applicable to the embodiment. FIG. 19 is a diagram for explaining camera position and orientation estimation processing applicable to the embodiment. FIG. 20 is an exemplary flowchart illustrating record creation processing according to the embodiment. FIG. 21 is a diagram showing an example of a report based on research report data according to the first embodiment. FIG. 22 is a flowchart of an example of record creation processing according to the first modification of the embodiment. FIG. 23 is a diagram showing an example of map data output according to the first modification of the embodiment. FIG. 24 is a flowchart of an example of record creation processing according to the second modification of the embodiment. FIG. 25 is an exemplary flowchart illustrating search processing for record data according to the second modification of the embodiment. FIG. 26 is a diagram illustrating a display example of search results according to the second modification of the embodiment. FIG. 27 is a diagram showing a display example of search results according to the second modification of the embodiment.

Exemplary embodiments of an information processing device, an information processing method, and a display device will be described in detail below with reference to the accompanying drawings.

[Overview of existing road property inspection methods]
Before describing the embodiments, an existing method for inspecting road properties will be briefly described. As an index for evaluating road properties, a pavement maintenance control index (MCI: Maintenance Control Index) is defined. MCI quantitatively evaluates the usability of pavement using three types of road surface property values: "crack rate", "rutting amount" and "flatness".

Among these, the "crack rate" is obtained by calculating the crack area according to the number of cracks and the patching area in each area in which the road surface is divided into 50 cm meshes, as follows, and the calculation result is expressed in formula (1). Apply to determine the crack rate.

1 crack ⇒ 0.15 m 2 crack ² cracks ⇒ 0.25 m ² crack Patching area 0 to 25% ⇒ 0 m ² crack
Patching area 25-75% ⇒ Crack 0.125m ²
Patching area 75% or more ⇒ Crack 0.25m ²

The "rut amount" is measured by measuring the depths D1 and D2 of _two "ruts" generated _per lane, as illustrated in Figs. 1(a) and 1(b). The larger of the depths D ₁ and D ₂ is taken. Since there are several patterns in how "ruts" are dug, a measurement method suitable for each pattern is defined. In FIG. 1(a), the distance between two "ruts" is higher than both ends of the two "ruts." It shows an example of the measurement method when it is lower than both ends of the rut.

The flatness is measured by measuring the height from the reference plane at three points at intervals of 1.5 m along the traveling direction of the vehicle on the road surface. For example, as shown in FIG. 2, three measuring instruments are provided on the underside of the vehicle at intervals of 1.5 m to measure heights X ₁ , X ₂ and X ₃ . Based on the measured heights X ₁ , X ₂ and X ₃ , the amount of displacement d is obtained by equation (2). This measurement is performed multiple times while moving the vehicle. Equation (3) is calculated based on the plurality of displacement amounts d thus obtained to calculate the flatness σ.

The "crack rate", "rutting amount" and "flatness" described above are measured for each 100 m evaluation section. If the location of damage is known in advance, 100m may be divided into smaller units such as 40m+60m for measurement. MCI is calculated based on the results of measurements performed in units of 100m, and a report is created.

MCI is calculated using the four formulas shown in Table 1 based on the measured crack rate C [%], rutting amount D [mm], and flatness σ [mm] to obtain the values MCI, MCI1, MCI2 and Calculate MCI3. Then, among the calculated values MCI, MCI1, MCI2 and MCI3, the minimum value is adopted as MCI. Evaluation is performed according to the evaluation criteria shown in Table 2 based on the adopted MCI, and it is determined whether or not the road surface in the evaluation section needs repair.

In addition, based on the measurement results, prepare a report summarizing the results of the road surface condition survey. 3 and 4 are diagrams showing examples of existing records. FIG. 3 shows an example of a record (referred to as record A) that displays the measured value for each 100 m, which is the measurement distance unit. FIG. 4 shows an example of a report (referred to as report B) that displays an image at a representative position and a measured value for each 100 m, which is the measurement distance unit.

In the example of FIG. 3, the report A contains the cumulative distance of the measurement section, the location and name of facilities within the measurement section, the coordinates (latitude, longitude) of the starting point and the end point of the measurement section, the section distance of the measurement section, and the date of inspection. Each item of date, inspection method, evaluation value and calculated value is included. Of these, the accumulated distance and section distance are displayed as values for every 100 m, for example. The evaluation values also include the crack rate C, the amount of rutting D, and the flatness σ, which are processed for each 100 m, which is the unit of measurement distance. In this example, the number of patches is also included in the evaluation value. The calculated values are the formula (1 = MCI, 2 = MCI1, 3 = MCI2, 4 = MCI3) used to calculate the MCI value and the MCI value at 100 m, which is the unit of measurement distance, and the longitudinal unevenness (IRI: International Roughness Index) is displayed.

As shown in FIG. 4, the report B includes each value of the section (cumulative distance), facilities, etc., crack rate C, rut amount D, flatness σ, MCI, and IRI for each 100 m unit of measurement distance. , and images at representative positions.

[Embodiment]
Next, an imaging system according to an embodiment will be described. In the embodiment, a camera capable of stereoscopic imaging (stereo camera) is attached to the vehicle to image the road surface. Depth information for the road surface is obtained from the imaging position based on the captured stereo images, the three-dimensional shape of the road surface is generated, and three-dimensional road surface data is created. By analyzing this three-dimensional road surface data, it is possible to obtain the "crack rate", "rutting amount" and "flatness" used to obtain the MCI.

More specific description will be given. A stereo camera is provided with two cameras separated by a predetermined length (referred to as a base line length), and outputs a pair of captured images (referred to as stereo captured images) captured by these two cameras. . By searching for a corresponding point between two captured images included in this stereo captured image, the depth distance can be restored for any point in the captured image. Data obtained by restoring the depth of the entire captured image and representing each pixel with depth information is called a depth map. That is, the depth map is three-dimensional point group information consisting of a set of points each having three-dimensional information.

This stereo camera is mounted downward at one location such as the rear of the vehicle so that it can take an image of the ground, and the vehicle is moved along the road to be measured. For the sake of explanation, it is assumed that the imaging range of the stereo camera mounted on the vehicle for measurement covers a prescribed length in the road width direction.

For the rutting amount D, the depth values of the striped portions cut in the road width direction in the depth map restored from the stereo image captured by the stereo camera are arranged. Based on this road width variation in depth, the rut depths D ₁ and D ₂ can be calculated.

The crack rate C is obtained by analyzing a captured image of the road surface to detect "cracks" and the like, and performing calculation using the above-described formula (1) based on the detection results.

Here, in the traveling direction of the vehicle, since the imaging range of one imaging is limited, for example, the crack rate C of a 100 m section cannot be calculated based on the imaging image obtained by one imaging. Therefore, while moving the vehicle along the road, images are sequentially captured for each movement corresponding to the length of the imaging range in the direction in which the vehicle travels. At this time, the imaging trigger is controlled so that the imaging range of the previous imaging and the imaging range of the current imaging overlap at a predetermined overlapping rate or more.

In this way, by controlling the imaging trigger according to the movement of the vehicle, the road surface to be measured can be imaged without omission. For this reason, captured images that are sequentially captured in accordance with the progress of the vehicle are joined using image processing called stitching, for example, to generate a single image including an image of the road surface in a 100 m section. By visually confirming or analyzing this image, the crack rate C on the road surface can be measured.

Flatness measurement uses a technique called Structure from Motion (SfM), which estimates the imaging position based on sufficiently overlapped images taken from images at different imaging points.

An outline of the SfM processing will be described. First, using images captured in overlapping imaging ranges, points obtained by imaging the same point in each image are detected as corresponding points. It is desirable to detect as many corresponding points as possible. Next, for example, simultaneous equations are established using the coordinates of the detected corresponding points for the movement of the camera from the imaging point of the first image to the imaging point of the second image, with rotation and translation as unknown parameters. , find the parameters that minimize the total error. In this way, the imaging position of the second image can be calculated.

As described above, the depth of an arbitrary point in the image can be restored as a depth map from the stereo-captured image at each imaging point. The depth map corresponding to the second stereo-captured image is obtained from the simultaneous equations described above with respect to the depth map corresponding to the first stereo-captured image, with the imaging position of the second camera as the origin. Coordinate transformation to map. As a result, the two depth maps can be unified into the coordinate system of the first depth map. In other words, one depth map can be generated by synthesizing two depth maps.

By performing this processing for, for example, depth maps based on all stereo imaged images captured in a 100m section, which is a measurement distance unit, and synthesizing the depth maps, the road surface of the 100m section is restored in one three-dimensional space. The amount of displacement d can be calculated by applying the depth value of the road surface thus restored to the above equation (2). Applying this displacement amount d to the above equation (3), the flatness σ is calculated.

In the embodiment, the flatness σ of the road surface, the amount of rutting D, and the crack rate for obtaining the MCI are obtained by imaging with a stereo camera mounted on a vehicle and performing image processing on the stereo-captured image. C can be measured collectively. In the embodiment, this measurement can be performed using an imaging system including a stereo camera and an information processing device that performs image processing on stereo captured images, and MCI can be obtained with a simple configuration.

In addition, with the existing technology, each of the three indicators for determining MCI is measured by separate devices, so control and management such as data storage and time synchronization between data are complicated. . On the other hand, in the embodiment, since the three indices for obtaining MCI are calculated based on a common stereo image, control and management such as data storage and time synchronization between data are facilitated.

[Camera Arrangement Applicable to Embodiment]
Next, an example of camera arrangement applicable to the embodiment will be described. FIG. 5 is a diagram illustrating a configuration example of an imaging system according to the embodiment; FIG. 5A is a side view of the vehicle 1 showing how the imaging system according to the embodiment is mounted on the vehicle 1. FIG. In FIG. 5( a ), the direction toward the left end of the drawing is the traveling direction of the vehicle 1 . That is, in FIG. 5( a ), the left end side of the vehicle 1 is the front portion of the vehicle 1 , and the right end side is the rear portion of the vehicle 1 . FIG. 5(b) is a diagram showing an example of the vehicle 1 viewed from the rear side.

In the imaging system according to the embodiment, a mounting member 3 having a mounting portion 2 for mounting an imaging device is fixed to the rear portion of a vehicle body 1 as a mobile body on which the imaging system is mounted, and one or more stereo cameras are mounted on the mounting portion 2. Install 6. Here, as illustrated in FIG. 5B, two stereo cameras 6L and 6R are attached to both ends of the vehicle body of the vehicle 1 in the width direction. Each of the stereo cameras 6L and 6R is attached in an orientation to image the road surface 4 on which the vehicle 1 moves. Preferably, each stereo camera 6L and 6R is mounted so as to image the road surface 4 from the vertical direction.

Hereinafter, the stereo cameras 6L and 6R will be collectively described as the stereo camera 6 when there is no need to distinguish between the stereo cameras 6L and 6R.

The stereo camera 6 is controlled by, for example, a personal computer (PC) 5 installed inside the vehicle 1, for example. The operator operates the PC 5 and instructs the stereo camera 6 to start imaging. When instructed to start imaging, the PC 5 starts imaging with the stereo camera 6 . The imaging is controlled in timing according to the moving speed of the stereo camera 6, that is, the vehicle 1, and is repeatedly executed.

The operator operates the PC 5 and instructs the end of imaging, for example, according to the end of imaging of a required section. The PC 5 terminates the imaging by the stereo camera 6 in response to the imaging termination instruction.

6 and 7 are diagrams showing examples of the imaging range of the stereo camera 6 applicable to the embodiment. In FIGS. 6 and 7, the same reference numerals are assigned to the parts common to FIG. 5 described above, and detailed description thereof will be omitted.

FIG. 6 is a diagram for explaining an imaging range (referred to as a traveling direction visual field Vp) of the vehicle 1 in the traveling direction by the stereo camera 6, which is applicable to the embodiment. In FIG. 6, the direction toward the left end of the drawing is the traveling direction of the vehicle 1, as in FIG. 1 described above. The traveling direction visual field Vp is determined according to the angle of view α of the stereo camera 6 and the height h of the stereo camera 6 with respect to the road surface 4, as shown in FIG.

FIG. 7 is a diagram for explaining the imaging range of the vehicle 1 in the road width direction by the stereo camera 6, which is applicable to the embodiment. FIG. 7 is a view of the vehicle 1 viewed from the rear side, and parts common to those in FIG.

In FIG. 7(a), the stereo camera 6L has two imaging lenses 6L _L and 6L _R. A line connecting the imaging lenses 6L _L and _6LR is called a base line, and its length is called a base line length. Similarly, the stereo camera 6R also has two imaging lenses 6R _L and 6R _R separated by the base line length, and is arranged so that the base line is perpendicular to the traveling direction of the vehicle 1 .

FIG. 7B shows an example of imaging ranges of the stereo cameras 6L and 6R applicable to the embodiment. In the stereo camera 6L, the imaging ranges 60L _L and _60LR of the imaging lenses 6L _L and _6LR are overlapped with a shift according to the base line length and the height h. Similarly for the stereo camera 6R, imaging ranges 60R _L and 60R _R respectively by the imaging lenses 6R _L and 6R _R are superimposed with a shift according to the baseline length and height h.

Hereinafter, unless otherwise specified, these imaging ranges 60L _L and _60LR and imaging ranges 60R _L and 60R _R are collectively referred to as stereo imaging ranges 60L and 60R, respectively. As shown in FIG. 7B, the stereo cameras 6L and 6R have stereo imaging ranges 60L and 60R, one end of the stereo imaging range 60L in the width direction of the vehicle 1 and the other width direction of the stereo imaging range 60R. are arranged so as to overlap with a predetermined overlap rate in the region 61 .

FIG. 8 is a diagram showing overlap in the traveling direction of the vehicle 1 of the stereo imaging range 60L by the stereo camera 6L, which is applicable to the embodiment. In FIG. 8, the left camera field of view V _L and the right camera field of view V _R respectively indicate the fields of view (imaging ranges) of the imaging lenses 6L _L and _6LR in the stereo camera 6L. Assume that the stereo camera 6L takes two images of the moving vehicle 1 . The stereo camera 6L takes an image of the stereo imaging range 60L (imaging ranges _60LL and _60LR ) for the first time, and moves the vehicle 1 in the traveling direction of the vehicle 1 with respect to the stereo imaging range 60L for the second time. The imaging of the stereo imaging range 60L' (imaging ranges 60L _L ' and 60L _R ') moved by a distance corresponding to the distance is performed.

At this time, the imaging timing of the stereo camera 6L is controlled so that the stereo imaging range 60L and the stereo imaging range 60L' overlap with the traveling direction visual field Vp at an overlap rate equal to or greater than a predetermined travel direction overlap rate Dr. do. In this way, by sequentially imaging the stereo imaging ranges 60L and 60L' along the time axis, it is possible to facilitate the process of joining two stereo imaging images obtained by imaging the stereo imaging ranges 60L and 60L'. .

Although the imaging system according to the embodiment uses the two stereo cameras 6L and 6R in the above description, the imaging system is not limited to this example. For example, as shown in FIG. 9A, the imaging system according to the embodiment adds one stereo camera 6C to the stereo cameras 6L and 6R, and adds three stereo cameras 6L, 6R and 6C. may be configured using

In the example of FIG. 9(a), the distance between the stereo cameras 6L and 6R is wider than when only the stereo cameras 6L and 6R are used, and the stereo camera 6C is placed in the center. As shown in FIG. 9B, imaging ranges 60C _L and 60C _R formed by imaging lenses 6C _L and 6C _R of the stereo camera 6C constitute a stereo imaging range 60C. The stereo cameras 6L, 6C and 6R are arranged so that the respective stereo imaging ranges 60L, 60C and 60R overlap in the width direction of the vehicle 1 at a predetermined overlap rate.

In this way, by using the three stereo cameras 6L, 6C and 6R to capture an image of one lane, it is possible to capture images by setting the image capturing ranges on the right side, the center and the left side of the lane, respectively, resulting in high image quality ( (high resolution) can be captured by a small number of stereo cameras. Here, especially the road width is generally defined as 3.5 m. Therefore, it is conceivable to image both ends of the lane in the road width direction with the stereo cameras 6L and 6R and image the central portion with the stereo camera 6C, corresponding to the road width of 3.5 m.

Note that the imaging system may be constructed using only one stereo camera having an angle of view capable of covering the road width (3.5 m) for which the imaging range is defined.

[Configuration of Imaging System According to Embodiment]
Next, the configuration of the imaging system according to the embodiment will be described. In the following description, it is assumed that the imaging system has two stereo cameras.

FIG. 10 is a block diagram showing an example of a schematic configuration of an imaging system according to the embodiment; 10, the imaging system 10 includes imaging units 100 ₁ and 100 ₂ , an imaging control unit 101, an information processing unit 102, a measurement data storage unit 103, a display control unit 104, a user input unit 105, and a display. and . A predetermined storage area of the storage 5004 can be applied to the measurement data storage unit 103, for example.

The imaging units 100 ₁ and 100 ₂ correspond to the stereo cameras 6L and 6R described above, respectively. The imaging control unit 101 controls imaging operations such as imaging timing, exposure, and shutter speed of the imaging units 100 ₁ and 100 ₂ . The information processing unit 102 generates three-dimensional road surface data including three-dimensional information of the road surface 4 based on each stereo captured image output from the imaging control unit 101 . The information processing section 102 stores the generated three-dimensional road surface data in the measurement data storage section 103 .

Further, the information processing unit 102 calculates each index such as the crack rate C, the rutting amount D, and the flatness σ based on the three-dimensional road surface data stored in the measurement data storage unit 103, and uses the calculated indexes. Ask for MCI. At this time, the information processing unit 102 obtains each index and MCI for each existing distance measurement unit of 100 m (referred to as a defined unit), and in a unit shorter than 100 m (referred to as a short unit), local Each index and MCI are calculated systematically. The short unit is 1 m, 5 m, 10 m, 20 m, etc., and can be set appropriately as long as the distance unit is less than 100 m. The information processing unit 102 can obtain each index and MCI for each of a plurality of short units. The information processing section 102 can store each index and MCI for each prescribed unit and each index and MCI for each short unit thus obtained in the measurement data storage section 103 .

The shortest length of the short unit is preferably 1 m. This is because there are damages that occur within a range of about 1 m, such as potholes and "hair cracks" around structures such as manholes. By making it possible to set the shortest unit from the shortest 1m, it is possible to grasp these damages without problems.

Further, the information processing section 102 creates a record based on each index and MCI. The created record is stored in the measurement data storage unit 103 . The information processing unit 102 creates a record based on each index and MCI for each prescribed unit (hereinafter referred to as a prescribed record) and a record based on each index and MCI for each short unit (hereinafter called a local record). can do.

A user input unit 105 accepts user input to an input device such as a keyboard, mouse, tablet, or touch panel. The display unit 106 includes a display device such as an LCD (Liquid Crystal Display) and a drive circuit that drives the display device, and causes the display device to display a screen according to a display control signal. The display control unit 104 generates a display control signal according to user input to the user input unit 105 and supplies the generated display control signal to the display unit 106 . For example, the display control unit 104 can read the record from the measurement data storage unit 103 and display it on the display unit 106 according to user input.

FIG. 11 is a block diagram showing an example hardware configuration of the imaging system 10 according to the embodiment. In FIG. 11, the imaging system 10 includes stereo cameras 6L and 6R, an information processing device 50 corresponding to the PC 5 in FIG.

Information processing device 50 generates a trigger at a predetermined timing and sends the generated trigger to stereo cameras 6L and 6R. The stereo cameras 6L and 6R take images in response to this trigger. Each stereo captured image captured by stereo cameras 6L and 6R is supplied to information processing device 50 . The information processing device 50 stores and accumulates the stereo captured images supplied from the stereo cameras 6L and 6R in a storage or the like. The information processing device 50 performs image processing such as generating a depth map and connecting the generated depth maps based on the accumulated stereo captured images to generate three-dimensional road surface data.

Further, the information processing device 50 calculates each index indicating the road surface properties such as the crack rate C, the amount of rutting D, and the flatness σ for each prescribed unit and each short unit based on the generated three-dimensional road surface data. The information processing device 50 obtains the MCI for each prescribed unit and the MCI for each short unit from the calculated indices.

The input device 51 is, for example, an input device such as a keyboard, mouse, or tablet. The display device 52 includes a display device such as an LCD and its drive circuit. The information processing device 50 calculates each index and MCI according to the user's input to the input device 51 and causes the display device 52 to display the record. The information processing device 50 can also output the record in a predetermined data format.

FIG. 12 is a block diagram showing an example configuration of the stereo camera 6L according to the embodiment. It should be noted that the stereo camera 6R can be realized with the same characteristics as the stereo camera 6L, so description thereof will be omitted here.

12, the stereo camera 6L includes imaging optical systems 600 _L and 600 _R , imaging elements 601 _L and 601 _R , driving units 602 _L and 602 _R , signal processing units 603 _L and 603 _R , and an output unit 604 and including. Among these, the imaging optical system 600 _L , the imaging device 601 _L , the driving unit 602 _L , and the signal processing unit 603 _L are configured to correspond to the imaging lens 6L _L described above. Similarly, the imaging optical system 600 _R , the imaging element 601 _R , the drive unit 602 _R , and the signal processing unit 603 _R are configured to correspond to the imaging lens 6L _R described above.

The imaging optical system _600L is an optical system having an angle of view α and a focal length f, and projects light from a subject onto an imaging element _601L . For example, it is an optical sensor using CMOS (Complementary Metal Oxide Semiconductor), and outputs a signal corresponding to projected light. An optical sensor using a CCD (Charge Coupled Device) may be applied to the imaging device _601L . The drive unit _602L drives the image sensor _601L , performs predetermined processing such as noise removal and gain adjustment on the signal output from the image sensor _601L , and outputs the processed signal. The signal processing unit _603L performs A/D conversion on the signal output from the driving unit _602L to convert the signal into a digital image signal (captured image). The signal processing unit 603 _L performs predetermined image processing such as gamma correction on the converted image signal and outputs the processed image signal. The captured image output from the signal processing unit 603 _L is supplied to the output unit 604 .

The operations of the imaging optical system 600 _R , the imaging device 601 _R , the driving unit 602 _R , and the signal processing unit 603 _R are similar to those of the imaging optical system 600 _L , the imaging device 601 _L , the driving unit 602 _L , and the signal processing unit described above. Since it is the same as _603L , the explanation here is omitted.

The driving units 602 _L and 602 _R are supplied with triggers output from the information processing device 50, for example. The driving units 602 _L and 602 _R take in signals from the imaging elements 601 _L and 601 _R according to this trigger and perform imaging.

Here, the driving units 602L and 602R perform the exposure in the imaging elements 601L and 601R by the batch simultaneous exposure method. This method is called a global shutter. On the other hand, since the rolling shutter takes in light in order from the top of the pixel position (line order), each line in the frame does not exactly capture the subject at the same time. In the case of the rolling shutter method, if the camera or the object moves at high speed while one frame of imaging signal is captured, the image of the object is picked up shifted according to the line position. Therefore, in the stereo cameras 6L and 6R according to the embodiment, a global shutter is used so that the road shape is captured correctly in terms of projection geometry.

The output unit 604 outputs the captured images of each frame supplied from the signal processing units 603 _L and 603 _R as a set of stereo captured images. The stereo captured images output from the output unit 604 are sent to the information processing device 50 and accumulated.

FIG. 13 is a block diagram showing an example configuration of an information processing apparatus 50 applicable to the embodiment. 13, the information processing device 50 includes a CPU (Central Processing Unit) 5000, a ROM (Read Only Memory) 5001, a RAM (Random Access Memory) 5002, and a graphics I/F (interface) connected to a bus 5030. ) 5003 , a storage 5004 , an input device 5005 , a data I/F 5006 and a communication I/F 5007 . Further, the information processing device 50 includes a camera I/F 5010 and a GNSS section 5021 which are connected to the bus 5030 respectively.

The storage 5004 is a storage medium that stores data in a nonvolatile manner, and can be a hard disk drive or flash memory. The storage 5004 stores programs and data for the CPU 5000 to operate.

The CPU 5000 uses the RAM 5002 as a work memory according to programs pre-stored in the ROM 5001 and the storage 5004 , for example, and controls the overall operation of the information processing apparatus 50 . The graphics I/F 5003 generates display signals that can be handled by the display 5020 based on display control signals generated by the CPU 5000 according to a program. A display 5020 displays a screen according to the display signal supplied from the graphics I/F 5003 .

The input device 5005 receives user operations and outputs control signals according to the received user operations. As the input device 5005, a pointing device such as a mouse or a tablet, or a keyboard can be applied. Further, the input device 5005 and the display 5020 may be integrally formed to form a so-called touch panel configuration.

A data I/F 5006 transmits and receives data to and from an external device. As the data I/F 5006, for example, a USB (Universal Serial Bus) is applicable. A communication I/F 5007 controls communication with an external network according to instructions from the CPU 5000 .

A camera I/F 5010 is an interface for each of the stereo cameras 6L and 6R. Each stereo picked-up image output from each stereo camera 6L and 6R is passed to CPU5000 via camera I/F5010. In addition, the camera I/F 5010 generates the above-described triggers according to instructions from the CPU 5000, and sends the generated triggers to the stereo cameras 6L and 6R.

The GNSS unit 5021 receives signals from (Global Navigation Satellite System) and acquires position information and speed information. The GNSS unit 5021 acquires speed information indicating the speed of the vehicle 1 based on the Doppler effect of the received GNSS signal. Note that the speed information can also be obtained directly from the vehicle 1 . When the stereo cameras 6L and 6R are attached to the vehicle 1, the speed information acquired by the GNSS unit 5021 indicates the speed of the stereo cameras 6L and 6R with respect to the subject (road surface).

FIG. 14 is an example functional block diagram for explaining functions of the information processing apparatus 50 applicable to the first embodiment. In FIG. 14 , the information processing apparatus 50 includes a captured image acquisition section 500 , a UI section 501 , a control section 502 , an imaging control section 503 and a position information acquisition section 504 . The information processing device 50 further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a state characteristic value calculation unit 521, a record processing unit 522, and a display information generation unit 523. include.

These captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, position information acquisition unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, The record processing unit 522 and the display information generation unit 523 are implemented by programs operating on the CPU 5000 . Not limited to this, the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, position information acquisition unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristics A part or all of the value calculation unit 521, the report processing unit 522, and the display information generation unit 523 may be configured by hardware circuits that operate in cooperation with each other.

The location information acquisition unit 504 acquires location information indicating the current location from the GNSS unit 5021 . The location information is obtained as latitude and longitude information, for example. The captured image acquisition unit 500 acquires stereo captured images from the stereo cameras 6L and 6R. In addition, the captured image acquiring unit 500 acquires position information indicating the current position by the position information acquiring unit 504 when acquiring the stereo captured image, and associates the acquired position information with the stereo captured image. The captured image acquisition unit 500 stores the acquired stereo captured image and position information in the storage 5004, for example. Also, the captured image acquisition unit 500 acquires the stored stereo captured image and position information from the storage 5004, for example.

The UI unit 501 implements a user interface by displaying on the input device 5005 and display 5020 . A control unit 502 controls the operation of the information processing apparatus 50 as a whole.

The imaging control unit 503 corresponds to the imaging control unit 101 described above. That is, the imaging control unit 503 acquires speed information indicating the speed of each of the stereo cameras 6L and 6R with respect to the subject (road surface 4), and combines the acquired speed information with the preset angle of view α of each of the stereo cameras 6L and 6R. , height h, and a trigger for instructing the imaging of each of the stereo cameras 6L and 6R.

The matching processing unit 510 performs matching processing using the two captured images forming the stereo captured image acquired by the captured image acquisition unit 500 . The 3D information generation unit 511 performs processing related to 3D information. For example, the 3D information generation unit 511 obtains depth information by trigonometry or the like using the result of matching processing by the matching processing unit 510, and generates 3D point group information based on the obtained depth information. Also, the 3D information generating unit 511 generates 3D road surface information from 3D point group information generated based on the stereo imaged images sequentially captured with overlapped portions in the traveling direction of the vehicle 1 .

The 3D information acquisition unit 520 acquires the 3D point cloud information obtained for each stereo image by the 3D information generation unit 511 . The state characteristic value calculation unit 521 uses each three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to calculate each index for obtaining MCI. , the state characteristic values of the crack rate C, the amount of rutting D, and the flatness σ are calculated. In other words, the state characteristic value calculation unit 521 can be said to be a conversion unit that converts each three-dimensional point group information, which is measurement data obtained by measuring the road surface, and each stereo image into each state characteristic value. At this time, the state characteristic value calculation unit 521 also acquires position information associated with the stereo image. The state characteristic value calculation unit 521 associates each calculated state characteristic value with the acquired position information, and stores them in the measurement data storage unit 103 (storage 5004).

The record processing unit 522 performs processing related to records. For example, the record processing unit 522 obtains MCI based on each state characteristic value calculated by the state characteristic value calculation unit 521, and creates record data including position information. The report processing unit 522 creates prescribed report data processed for each prescribed unit and local report data processed for each short unit, and stores them in the measurement data storage unit 103, for example. Further, the record processing unit 522 searches for prescribed record data and local record data according to user instructions, and outputs search results.

The display information generation unit 523 generates display information for displaying the record output by the record processing unit 522, for example, according to an instruction from the UI unit 501 according to user input. The display information generation unit 523 also generates display information for displaying a screen based on each state characteristic value calculated by the state characteristic value calculation unit 521, according to an instruction from the UI unit 501 according to a user input.

A program for realizing each function according to the embodiment in the information processing apparatus 50 is a file in an installable format or an executable format and stored on a CD (Compact Disk), a flexible disk (FD), a DVD (Digital Versatile Disk), or the like. provided on a computer-readable recording medium. Alternatively, the program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. Also, the program may be configured to be provided or distributed via a network such as the Internet.

The program includes a captured image acquisition unit 500, a UI unit 501, a control unit 502, an imaging control unit 503, a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a state characteristic value calculation unit 521, and a record processing unit. 522 and a display information generation unit 523 are included. As actual hardware, the CPU 5000 reads out the program from a storage medium such as the storage 5004 and executes it, thereby loading each of the above-described units onto the main storage device such as the RAM 5002 . , control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, record processing unit 522, and display information generation unit 523 are stored on the main storage device. generated.

[Trigger Generation Method According to Embodiment]
Next, a method of generating triggers for instructing imaging to the stereo cameras 6L and 6R according to the embodiment will be described in more detail. In the embodiment, the imaging processing unit 101 moves the distance in the direction of the speed of the vehicle 1 in the stereo imaging range of the object to be measured (road surface 4) of the stereo cameras 6L and 6R at the speed indicated by the speed information. Triggers are generated at time intervals equal to or less than a predetermined time shorter than the time.

That is, the trigger needs to be generated so that the imaging range of the road surface 4 by the stereo cameras 6L and 6R maintains a predetermined overlapping rate (advancing direction overlapping rate Dr) in the traveling direction of the vehicle 1 . As will be described later, the purpose of this is to detect sufficient corresponding points in order to stably and accurately calculate the camera position in the process of calculating the imaging position from each stereo imaged image. The lower limit of the traveling direction overlap rate Dr is experimentally determined, for example, as "60%".

In the embodiment, the following three methods can be applied as a trigger generation method.
(1) Method of generating at regular time intervals (first generation method)
(2) A method of generating by detecting the movement speed of the camera (second generation method)
(3) A method of calculating and generating a movement distance using a captured image (third generation method)

(First generation method)
First, the first trigger generation method will be described. In the first generation method, the trigger time interval is determined from the maximum speed Speed of the vehicle 1 during imaging and the size of the imaging range (the length of the imaging range in the traveling direction of the vehicle 1). The imaging control unit 101 acquires in advance the maximum speed Speed of the vehicle 1 based on system setting values in the vehicle 1 and user input to the information processing device 50 . The imaging control unit 503 calculates the trigger time interval using the acquired maximum speed Speed, the traveling direction visual field Vp, and the traveling direction overlapping rate Dr, using the following equation (4). Note that the above-described lower limit value is applied to the traveling direction overlapping rate Dr.

The number of triggers fps to be generated per second is calculated by the formula (4). The reciprocal of the number of triggers fps is the time interval until the next trigger to be generated.

As described with reference to FIG. 6, the traveling direction field of view Vp is schematically defined by the height h of each of the stereo cameras 6L and 6R from the road surface 4 and the angle of view α of each of the stereo cameras 6L and 6R. can be set based on In practice, furthermore, the angles of the stereo cameras 6L and 6R with respect to the road surface 4 are taken into consideration when setting the traveling direction visual field Vp.

Here, when the moving vehicle 1 curves to the right or left, the amount of movement of the outer camera out of the stereo cameras 6L and 6R increases. Therefore, it is more preferable to use the speed of rotational movement based on the position of the outer camera instead of using the maximum speed Speed of the vehicle 1 as it is.

The imaging control unit 503 stores and accumulates all the stereo imaged images imaged according to the generated trigger in the storage 5004 or the RAM 5002, for example.

(Second generation method)
Next, a second trigger generation method will be described. While the above-described first method is simple, when the vehicle 1 is stopped or moving at a speed lower than a predetermined speed, images are captured at excessively fine intervals with respect to the maximum speed Speed. As a result, the amount of stereo imaged images to be accumulated becomes large. In the second generation method, the imaging control unit 503 detects the moving speed of the camera and generates a trigger according to the detected moving speed.

The imaging control unit 503 uses, for example, the current speed of the vehicle 1 indicated by the speed information acquired by the GNSS unit 5021 as the maximum speed Speed in Equation (4), calculates the time interval until the next trigger, and performs imaging. I do. According to the second generation method, the slower the moving speed of the vehicle 1 is, the longer the time interval of trigger generation is, thereby suppressing wasteful imaging.

The imaging control unit 503 stores and accumulates all the stereo imaged images imaged according to the generated trigger in the storage 5004 or the RAM 5002, for example.

Note that this second generation method and the above-described first generation method can be implemented in combination.

(Third generation method)
Next, a third trigger generation method will be described. In the third generation method, as in the first generation method described above, triggers are generated at regular time intervals based on the maximum speed Speed of the vehicle 1 . Here, in the third generation method, not all the stereo images captured in response to the trigger are accumulated, but only when the traveling direction overlapping rate Dr is less than a preset value.

As will be described later, it is possible to calculate the moving distance of the camera (vehicle 1) using only the stereo captured images captured so as to overlap with the traveling direction of the vehicle 1 .

The imaging control unit 503 calculates the moving distance of the camera using the last accumulated stereo image and the immediately preceding stereo image. The imaging control unit 503 determines whether or not the calculated distance exceeds the movement distance corresponding to the lower limit of the traveling direction overlapping rate Dr. When the image capturing control unit 503 determines that the time limit has been exceeded, the image capturing control unit 503 accumulates the stereo image captured immediately before. On the other hand, if it is determined that the imaging control unit 503 does not exceed the threshold, the imaging control unit 503 discards the stereo image captured immediately before.

As a result, even without using a sensor for measuring the current speed of the vehicle 1, wasteful image accumulation is not performed when the moving speed is low.

[Method for Calculating Road Condition Value According to Embodiment]
Next, a method for calculating road surface property values according to the embodiment will be described. A method for measuring the flatness σ, the amount of rutting D, and the crack ratio C according to the embodiment will be described below.

(flatness)
FIG. 15 is an exemplary flowchart illustrating flatness calculation processing according to the embodiment. In step S100, the captured image acquisition unit 500 acquires stereo captured images captured by the stereo cameras 6L and 6R and stored in the storage 5004, for example. When the stereo imaged images are acquired, the process moves to steps S101a and S101b, which can be processed in parallel.

In step S101a, a depth map is generated based on the stereo captured image acquired in step S100. Depth map generation processing applicable to the embodiment will be described with reference to FIGS. 16 and 17 . FIG. 16 is an exemplary flowchart illustrating depth map generation processing applicable to the embodiment.

In step S<b>120 , the matching processing unit 510 acquires stereo captured images from the captured image acquisition unit 500 . In the next step S121, the matching processing unit 510 performs matching processing based on the two captured images forming the acquired stereo captured image. In the next step S122, the 3D information generation unit 511 calculates depth information based on the matching processing result of step S121, and generates a depth map, which is three-dimensional point cloud information.

The processing of steps S121 and S122 will be described more specifically. In the embodiment, depth information is calculated by a stereo method using two captured images that constitute a stereo captured image. The stereo method here uses two captured images captured from different viewpoints by two cameras, and for a certain pixel (reference pixel) in one captured image, a corresponding pixel (reference pixel) in the other captured image ( In this method, the corresponding pixel) is obtained, and the depth (depth value) is calculated by triangulation based on the reference pixel and the corresponding pixel.

In step S121, the matching processing unit 510 uses the two captured images constituting the stereo captured image, which are acquired from the captured image acquisition unit 500, and a predetermined size centering on the reference pixel in one captured image serving as a reference. A search is performed by moving a region corresponding to the region in the other captured image to be searched within the other captured image.

Various methods are known for searching for corresponding pixels, and for example, a block matching method can be applied. The block matching method acquires pixel values of a region cut out as a block of M pixels×N pixels centering on a reference pixel in one captured image. Also, in the other captured image, a pixel value of a region cut out as a block of M pixels×N pixels centering on the target pixel is acquired. Based on the pixel values, the degree of similarity between the area containing the reference pixel and the area containing the target pixel is calculated. The similarity is compared while moving blocks of M pixels×N pixels in the search target image, and the target pixel in the block at the position where the similarity is highest is taken as the corresponding pixel corresponding to the reference pixel.

Similarity can be calculated by various calculation methods. For example, the normalized cross-correlation (NCC) shown in Equation (5) is one of the cost functions, and the higher the value of the numerical value C _NCC indicating the cost, the higher the similarity. indicates In equation (5), the values M and N represent the size of the pixel block to search. Also, the value I(i, j) represents the pixel value of a pixel in a pixel block in one of the reference captured images, and the value T(i, j) represents the pixel block in the other captured image to be searched. represents the pixel value in

As described above, the matching processing unit 510 moves the pixel block in the other captured image corresponding to the pixel block of M pixels×N pixels in the other captured image, for example, in units of pixels. Calculate the numerical value C _NCC by performing the calculation of equation (5). In the other captured image, the center pixel of the pixel block with the largest numerical value C _NCC is set as the target pixel corresponding to the reference pixel.

Returning to the description of FIG. 16, in step S122, the 3D information generation unit 511 calculates a depth value (depth information) using trigonometry based on the reference pixels and corresponding pixels obtained by the matching processing in step S121. , generate three-dimensional point cloud information relating to one captured image and the other captured image that constitute a stereo captured image.

FIG. 17 is a diagram for explaining trigonometry applicable to the embodiment; Calculating the distance S to the target object 403 (for example, one point on the road surface 4) in the figure from the imaging position information in the image captured by each imaging element 402 (corresponding to the imaging elements 601 _L and 601 _R ). is the purpose of processing. That is, this distance S corresponds to the depth information of the target pixel. The distance S is calculated by Equation (6) below.

In equation (6), the value baseline represents the length of the baseline (baseline length) between the cameras 400a and 400b. In the example of FIG. 7, this corresponds to the baseline length by the imaging lenses 6L _L and 6L _R (in the case of the stereo camera 6L). The value f represents the focal length of the lens 401 (corresponding to the imaging lenses 6L _L and 6L _R ). The value q represents parallax. The parallax q is a value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel by the pixel pitch of the imaging element. The coordinate values of the corresponding pixels are obtained based on the result of matching processing in step S121.

This formula (6) is a method of calculating the distance S when the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R are used. This is to calculate the distance S from the captured images captured by the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R , respectively. In the embodiment, the calculation method based on this equation (6) is applied to each captured image captured by the imaging lenses 6L _L and _{6LR of the stereo camera 6L and the imaging lenses 6RL and 6R R of the} stereo camera 6R. , the distance S is calculated for each pixel.

Returning to the description of FIG. 15, in step S101b, the 3D information generation unit 511 estimates the camera position and orientation. Here, the camera position indicates, for example, the coordinates of the center of the stereo camera 6L when viewed as one unit. Details of the processing in step S101b will be described later.

When the processing of steps S101a and S101b ends, the processing moves to step S102. In step S102, the 3D information generating unit 511 converts two depth maps that are adjacent in time series to match the coordinate system of the depth map at the previous time based on the camera position and orientation estimated in step S101b. Coordinate transformation of the depth map of the next time after the time is performed.

Note that the previous time and the next time are the first imaging and the second imaging that are continuously executed in chronological order in response to a trigger in the stereo cameras 6L and 6R. The time is the previous time, and the time when the second imaging is performed is the next time. That is, the depth map at the previous time is a depth map generated based on the stereo image captured by the first imaging, and the depth map at the next time is a depth map generated based on the stereo image captured by the second imaging. be.

In the next step S103, the 3D information generation unit 511 integrates the depth map of the next time coordinate-transformed in step S102 into the depth map of the previous time. That is, the two depth maps of the previous time and the next time are arranged in a common coordinate system by the coordinate conversion in step S102, so that these two depth maps can be integrated. The processing of steps S102 and S103 is performed on stereo captured images captured at all times.

As shown in FIGS. 7 and 9, when a plurality of stereo cameras 6L and 6R or stereo cameras 6L, 6C and 6R are arranged side by side in the road width direction, the stereo cameras arranged in the road width direction (for example, stereo cameras) In the cameras 6L and 6R) as well, the camera position and orientation are similarly estimated, and depth map integration is then performed.

Further, the moving distance of the camera (vehicle 1) in the above-described third generation method can be calculated in the coordinate conversion in step S102 and the depth map integration processing in step S103.

In the next step S104, the 3D information acquisition unit 520 acquires the depth map integrated in step S103 from the 3D information generation unit 511. FIG. Then, the state characteristic value calculator 521 calculates the flatness σ based on the depth map acquired by the 3D information acquirer 520 . That is, when the depth maps are integrated, the road surface shape of the road in the measured section is restored as a point cloud in one three-dimensional space. When the point cloud is restored, for each sampling point on the road surface, take a coordinate system that connects the coordinates of the 1.5m points before and after according to the regulations, and calculate the distance to the 3D point cloud at the center, using the formula The amount of displacement d can be calculated in the same manner as in (2), and the flatness σ can be calculated from the amount of displacement d according to equation (3).

(rutting amount)
Regarding the rutting amount D, in step S104, the state characteristic value calculation unit 521 scans the depth map acquired by the 3D information acquisition unit 520 in the road width direction to obtain the values shown in FIGS. ), the depths D ₁ and D ₂ can be obtained. _The rutting amount D can be obtained based on the obtained depths D1 and D2 and the cross _- sectional information obtained by scanning the depth map.

(crack rate)
For example, the state characteristic value calculation unit 521 applies the stereo captured image acquired in step S100 to the depth map integrated in step S103. That is, the state characteristic value calculation unit 521 integrates each stereo captured image that is overlapped in the direction of travel of the vehicle 1 with the rate of overlap of the direction of travel Dr. The state characteristic value calculation unit 521 sets a mesh of 50 cm according to regulations for the integrated image, acquires information such as cracks and patching in each mesh by image analysis, and calculates the crack rate C according to regulations based on the acquired information. Calculate

(Processing for estimating camera position and orientation)
Next, the estimation processing of the camera position and direction in step S101b in the flowchart of FIG. 15 described above will be described in more detail. FIG. 18 is an example flowchart illustrating camera position and orientation estimation processing applicable to the embodiment. In embodiments, the camera position and orientation are estimated using the SfM described above.

In step S<b>130 , the 3D information generation unit 511 acquires stereo captured images from the captured image acquisition unit 500 . Here, the 3D information generation unit 511 acquires two stereo captured images (a stereo captured image at the previous time and a stereo captured image at the next time) continuously captured in time series. Next, in step S131, the 3D information generation unit 511 extracts feature points from each acquired stereo image. This feature point extraction process is a process of finding points that can be easily detected as corresponding points in each stereoscopic image. do.

In the next step S132, the 3D information generation unit 511 detects, from the stereo image of the next time, the point captured at the same place as the point extracted as the feature point in the stereo image of the previous time. For this detection processing, a technique called optical flow or a technique called feature point matching represented by SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), and the like can be applied.

At the next step S133, the 3D information generation unit 511 estimates the initial position and orientation of the camera. The 3D information generating unit 511 fixes the coordinates of the corresponding points detected in each stereoscopic image in step S132, and solves the simultaneous equations using the position and orientation of the camera that captured the stereoscopic image at the next time as parameters. , the position of the camera is estimated, and the three-dimensional coordinates are calculated (step S134).

A method for estimating the initial position and orientation of the camera will be described with reference to FIG. Equation (7) expresses the relationship between the coordinates x _ij of the spatial point X _j projected onto the camera (viewpoint) P _i . In equation (7), the value n represents the number of points in the three-dimensional space, and the value m represents the number of cameras (captured images).

Each value P is a projection matrix that transforms the coordinates of a point in the three-dimensional space into the two-dimensional coordinates of the image for each camera, and is expressed by the following equation (8). Equation (8) consists of a transformation matrix of 2 rows and 2 columns of three-dimensional coordinates shown on the right side, and a projective transformation f _i from the three-dimensional coordinates to the two-dimensional coordinates.

In FIG. 19, the values X _j and P _i are calculated using simultaneous equations given only the coordinates x _ij captured by the camera. A linear least-squares method can be used to solve this simultaneous equation.

In the next step S135, the 3D information generation unit 511 optimizes the three-dimensional coordinates indicating the camera position calculated in step S134. The position and orientation of the camera calculated by solving the system of equations may not have sufficient accuracy. Therefore, the accuracy is improved by performing the optimization process using the calculated value as the initial value.

The two-dimensional coordinates x _ij (reprojection (called coordinates) and the difference is called the residual. For all corresponding points in all stereoscopic images used to calculate the position and orientation of the camera, the parameters are adjusted by an optimization operation so that the sum of residual errors is minimized. This is called bundle adjustment. Global optimization by bundle adjustment can improve the accuracy of camera position and orientation.

The above is the base processing of camera position and orientation estimation in normal SfM. In the embodiment, the stereo cameras 6L and 6R with known base lengths are used. It is easy to search for corresponding points in two captured images forming a captured image. Therefore, the three-dimensional coordinates X _j in space are determined on an actual scale (actual size), and the position and orientation of the camera after movement can also be stably calculated.

As described above, in the imaging system 10 according to the embodiment, the information processing device 50 generates triggers, and distributes the generated triggers to all the stereo cameras 6L and 6R. and 6R synchronously perform imaging. At this time, clock generators of the stereo cameras 6L and 6R, trigger distribution components for distributing triggers in the information processing device 50, differences in trigger wiring lengths in the information processing device 50 and the stereo cameras 6L and 6R, etc. Due to the influence, there is a possibility that a slight deviation may occur in the timing at which the stereo cameras 6L and 6R actually capture the trigger and perform imaging. It is preferable to suppress this imaging timing lag as much as possible.

Especially between the stereo cameras 6L and 6R whose baseline lengths are known, it is desirable to minimize the deviation of the imaging timing. For example, when wiring the trigger supply from the camera I/F 5010 to the stereo cameras 6L and 6R, there are cases where a repeater is used to ensure trigger quality, and a photocoupler is used to protect against electrostatic noise. be. In this case, it is desirable to share these repeaters and photocouplers between the stereo cameras 6L and 6R.

[Record creation process according to the embodiment]
Next, the record creation process according to the embodiment will be described more specifically. FIG. 20 is an exemplary flowchart illustrating record creation processing according to the embodiment. When preparation of the report data is started, in step S200, the report processing unit 522 acquires and outputs the elements to be measured and calculated according to the regulations, and prepares the prescribed report data. Here, the elements to be measured and calculated according to the regulations are each evaluation value processed for each regulation unit (crack rate C, rutting amount D, flatness σ, etc.) and the calculated value obtained based on each evaluation value ( IRI, MCI value, MCI formula selection value).

The record processing unit 522 applies these measurement and calculation target elements to, for example, a predetermined format to create record data of the prescribed record. The record data created here are equivalent to the record A and record B described in FIGS. 5 and 6, for example. For the image included in the record B, for example, an image acquired by a drive recorder mounted on the vehicle 1 can be applied.

In the next step S201, the work paper processing unit 522 acquires the local measurement and calculation target elements from the measurement data storage unit 103, and creates local work paper data. Here, the local measurement and calculation target elements are each evaluation value (crack rate C, rutting amount D, flatness σ, etc.) processed for each short unit (5 m, 10 m, 20 m, etc.) and each evaluation value including calculated values (IRI, MCI value, MCI formula selection value, etc.) determined based on Each evaluation value is locally processed local crack area ratio C _L , local rutting amount D _L , and local flatness σ _L . Also, the calculated value is the local MCI _L . Of these, the local rutting amount D _L is based on the data at the central position of the short unit.

FIG. 21 is a diagram showing an example of the report A based on the local report data output in step S201 according to the first embodiment. Here, the explanation is given assuming that the short unit is 10m. In FIG. 21, the cumulative value of the distance in the section item is set to 10 m units of short units, and the section distance is set to 10 m. Also, each position information, each evaluation value, and each calculation value use values processed every 10 m. A record B can also be created in the same way.

The information processing apparatus 50 according to the embodiment thus outputs the crack rate C, the amount of rutting D, and the flatness σ for each prescribed unit when preparing the report, and the MCI obtained based on these, In addition to creating prescribed report data, the crack rate C, rutting amount D, and flatness σ for each short unit, and the MCI obtained based on these are output to create local report data. Therefore, it becomes easy to extract local information from a huge amount of imaging data and measurement data.

[First Modification of Embodiment]
Next, the 1st modification of embodiment is demonstrated. FIG. 22 is a flowchart of an example of record creation processing according to the first modification of the embodiment. In FIG. 22, the same reference numerals are assigned to the processes that are common to the processes in the flowchart of FIG. 20 described above, and detailed description thereof will be omitted. In steps S200 and S201, the report processing unit 522 acquires and outputs measurement and calculation target elements according to regulations, creates specified report data, acquires and outputs local measurement and calculation target elements, in the same manner as described above. and create local report data.

In the next step S210, the record processing unit 522 outputs map data corresponding to the measured section. The measurement data includes latitude and longitude information. As described above, the captured image acquiring unit 500 acquires the position information indicating the current position by the position information acquiring unit 504 when acquiring the stereo captured images from the stereo cameras 6L and 6R. Associate with the stereo captured image. The record processing unit 522 outputs map data based on this position information, and associates the position information with the position on the map data. Here, it is assumed that the record processing unit 522 has in advance map data of the area corresponding to the section to be measured.

FIG. 23 is a diagram showing an example of map data output according to the first modification of the embodiment. In the example of FIG. 23, a route 701 that has been measured on a map image 700 based on map data, that is, a stereo image taken while moving the vehicle 1 is highlighted. Further, the route 701 is displayed in the map image 700 by color-coding according to the MCI value, as shown in the legend at the bottom of FIG. This makes it possible to intuitively grasp road surface properties.

In FIG. 23, the route 701 is color-coded according to the MCI value, but this is not limited to this example. That is, the criteria for changing the display of the path 701 are not limited to the MCI value, but the crack rate C, the amount of rutting D, the flatness σ, the crack area ratio, the amount of local rutting C _L , the local flatness σ _L and the local MCI. You can choose from _L.

[Second Modification of Embodiment]
Next, the 2nd modification of embodiment is demonstrated. FIG. 24 is a flowchart of an example of record creation processing according to the second modification of the embodiment. In FIG. 24, the same reference numerals are assigned to the same processes as those in the flowchart of FIG. 22, and detailed description thereof will be omitted. In steps S200 and S201, the report processing unit 522 acquires and outputs measurement and calculation target elements according to regulations, creates specified report data, acquires and outputs local measurement and calculation target elements, in the same manner as described above. and create local report data. In step S210, the record processing unit 522 outputs map data corresponding to the measured section.

In the next step S220, the record processing unit 522 performs event analysis and outputs the analysis result. The document processing unit 522 analyzes possible locations of longitudinal cracks, transverse cracks, and dense cracks by event analysis, for example, using the local crack rate C _L . Here, if there is only one crack and n or more detected meshes are arranged in the vertical direction (traveling direction of the vehicle 1), there is a possibility of longitudinal cracks, n in the horizontal direction (road width direction). If the cracks are aligned as above, it is considered to be a possible transverse crack. In addition, when two or more cracks are detected and n×n meshes are clustered together, it is regarded as a possible location of dense cracks.

It is assumed that the value n for judging longitudinal cracks, transverse cracks, and dense cracks is set in advance. Also, the value n can be changed during subsequent retrieval and display.

In the second modification of the embodiment, the crack rate C is processed in units of distance shorter than the prescribed unit of distance measurement. Therefore, more detailed analysis can be easily performed.

Next, search processing according to the second modification of the embodiment will be described. FIG. 25 is an exemplary flowchart illustrating search processing for record data according to the second modification of the embodiment. In step S300, the report processing unit 522 acquires the measurement and calculation target elements (crack rate C, rutting amount D, flatness σ, MCI value, etc.) in accordance with the regulations in step S200, and creates prescribed report data. Display based on the prescribed record data. Map information corresponding to the position information included in the prescribed record data may be displayed together with the prescribed record data.

Here, it is assumed that a search instruction is input by the user in accordance with the display of the record data and the map information in step S300 (step S301). The search instruction includes the setting of the length of the short unit (5m, 10m, 20m, etc.), the setting of the value n for the event analysis and detection of possible locations described in step S220, the setting of search parameters, and the like. Examples of search parameters include local crack area ratio C _L , local flatness σ _L values, transverse cracks, longitudinal cracks, display of potential locations for dense cracks, and the like. These search parameters may be used alone, or may be obtained by logical sum or logical product of multiple search parameters.

In the next step S302, the report processing unit 522 outputs the search result corresponding to the search instruction in step S301 in one or more formats of, for example, report A (see FIG. 21), report B, and map information (see FIG. 23). Output and display on display 5020 . Here, the output data is the data of the section matching the search parameter set by the search instruction input in step S301. The data display order can be selected, for example, numerically ascending order or descending order.

It is assumed that a local data display instruction is input by the user in accordance with the display of the search results in step S302 (step S303). For example, the record processing unit 522 is instructed by the user's input to indicate a section for which data is to be displayed and the display content, which are included in the search result.

In the next step S304, the display information generation unit 523 generates display information for displaying a screen according to the local data display instruction input in step S303. For example, when the local data display instruction is an instruction to display the report B and the map information in the short unit section instructed for the search result, the display information generation unit 523 generates the position information, the evaluation Display information for displaying the report B and the map information is generated based on the value and the calculated value. The report B and the map information are displayed on the display 5020 according to this display information.

In the following, in order to avoid complexity, it is assumed that the display information generation unit 523 generates display information, and that a predetermined screen is displayed on the display 5020 according to the generated display information. is displayed, and so on.

Furthermore, as illustrated in FIG. 26, the display information generation unit 523 generates an image 820 of a short unit section based on the search result on a display area 800 of the display 5020, and displays a road surface of a section of 100 m, which is a prescribed unit. Along with the image 810, the position of the 100 m road surface in the image 810 is displayed (indicated by section 820'). Also, in this image 820, as shown in the mesh portion 821, the number of cracks in the 50 cm mesh (displayed in the upper right corner of the mesh portion 821) is displayed. In addition to the display of FIG. 26, the value of the local rutting amount D _L and the value of the local flatness σ _L in the section of the image 820 can be displayed.

Furthermore, the display information generation unit 523 can graphically display the value of the local rutting amount D _L and the value of the local flatness σ _L on the display of FIG. 26 . For example, as illustrated in FIG. 27, the display information generation unit 523 generates superimposed displays 8200 to 8204 on the image 820 according to the values of the local rutting amount D _L and the local flatness σ _L A transparency image can be superimposed. Not limited to this, it is also possible to change the gradation of the display of the image 820 according to the value of the local rutting amount D _L and the local flatness σ _L . At the same time, it is also possible to further display the result of the event analysis obtained in step S220 of the flowchart of FIG. 24 and the information on the possible locations.

On the screen displayed in step S304, it is possible to input memos such as confirmation items (step S305). For example, it is possible to input event analysis confirmation information through visual observation of images, information related to repair priority, other remark information, and the like. The record processing unit 522 can store the input memo information in association with the record in response to a user-inputted save instruction.

As described above, in the second modified example of the embodiment, it is possible to search and display record data in short units, so that more detailed evaluation of road surface properties can be performed.

Although each of the above-described embodiments is a preferred embodiment of the present invention, it is not limited thereto, and various modifications can be made without departing from the scope of the present invention.

1 vehicle 4 road surface 5 PC
6, 6L, 6R stereo cameras 6L _L , 6L _R , 6R _L , 6R _R imaging lens 10 imaging system 50 information processing devices 60C, 60L, 60R stereo imaging ranges 60C _L , 60C _R , 60L _L , 60L _R , 60R _L , 60R _R imaging range 100 ₁ , 100 ₂ imaging units 101 and 503 imaging control unit 103 generation unit 500 captured image acquisition unit 510 matching processing unit 511 3D information generation unit 520 3D information acquisition unit 521 state characteristic value calculation unit 522 record processing unit 523 Display information generation unit 700 Map image 5021 GNSS unit

Japanese Patent No. 4848532

Claims (10)

an acquisition unit that acquires measurement data of the road surface condition according to the direction of travel of the vehicle on the road surface;
a conversion unit that converts the measurement data into a state characteristic value that indicates the state of the road surface in units of a predetermined distance with respect to the vehicle traveling direction of the road surface;
a creation unit that creates a record based on the state characteristic value;
a display control unit that causes a display unit to display a screen based on the state characteristic value;
with
The condition characteristic value includes each index indicating crack rate, rutting amount, and flatness, which are used to calculate the pavement maintenance index,
The acquisition unit
Based on a plurality of stereo camera-captured images of the road surface captured at different times using a stereo camera , a plurality of 3 indicating depth information of the road surface corresponding to the plurality of stereo camera-captured images as the measurement data. Get the dimensional point cloud information,
The conversion unit
Based on the three-dimensional point cloud information obtained by integrating the plurality of three-dimensional point cloud information in a common coordinate system and the image captured by the stereo camera, the distance unit corresponding to the prescribed first distance unit with respect to the traveling direction of the vehicle on the road surface Information for conversion into a first state characteristic value, which is the state characteristic value, and a second state characteristic value, which is the state characteristic value corresponding to a second distance unit having a shorter distance than the first distance unit. processing equipment. The creation unit
2. The information processing apparatus according to claim 1, wherein a first report based on said first state characteristic value and a second report based on said second state characteristic value are created. The display control unit
Displaying information indicating a position corresponding to the second state characteristic value included in the second display based on the second state characteristic value in response to the first display based on the first state characteristic value. 3. The information processing apparatus according to claim 1 or 2. The display control unit
A screen based on the second condition characteristic value corresponding to the index indicating at least one of the rutting amount and the flatness is compared with the screen based on the second condition characteristic value corresponding to the index indicating the crack rate. 4. The information processing apparatus according to any one of claims 1 to 3, wherein the measurement positions are associated with each other, superimposed, and displayed on the display unit. The measurement data includes coordinate information indicating the coordinates of the point where the measurement was performed,
The display control unit
3. The information processing apparatus according to claim 1, wherein the display indicating the second state characteristic value is displayed on the display unit in association with map information based on the coordinate information. The condition characteristic value is a pavement maintenance index,
The display control unit
6. The information processing apparatus according to claim 5, wherein the display indicating the second state characteristic value is displayed in association with the map information using a color corresponding to the value of the pavement maintenance index. further comprising a storage unit that stores the converted state characteristic value,
The condition characteristic value includes each index indicating crack rate, rutting amount, and flatness, which are used to calculate the pavement maintenance index,
a search unit that searches the second state characteristic value stored in the storage unit using at least one of the crack rate, rutting amount, and flatness as a parameter;
The display control unit
7. The information processing apparatus according to any one of claims 1 to 6, wherein the display unit displays a screen based on the second state characteristic value searched by the search unit. The condition characteristic value includes an index indicating a crack rate used to calculate a pavement maintenance management index,
The information processing apparatus according to any one of claims 1 to 7, further comprising an analysis unit that analyzes a crack state based on the crack rate. The conversion unit
9. The information processing apparatus according to any one of claims 1 to 8, wherein the unit of distance is converted into a state characteristic value of less than 100 meters based on the integrated three-dimensional point cloud information and the image captured by the stereo camera. . an acquisition step of acquiring measurement data of the road surface condition according to the direction of travel of the vehicle on the road surface;
a conversion step of converting the measurement data into a state characteristic value indicating the state of the road surface in units of a predetermined distance with respect to the vehicle traveling direction of the road surface;
a creating step of creating a report based on the state characteristic value;
a display control step of displaying a screen based on the state characteristic value on a display unit;
has
The condition characteristic value includes each index indicating crack rate, rutting amount, and flatness, which are used to calculate the pavement maintenance index,
The obtaining step includes:
Based on a plurality of stereo camera-captured images of the road surface captured at different times using a stereo camera , a plurality of 3 indicating depth information of the road surface corresponding to the plurality of stereo camera-captured images as the measurement data. Get the dimensional point cloud information,
The conversion step includes:
Based on the three-dimensional point cloud information obtained by integrating the plurality of three-dimensional point cloud information in a common coordinate system and the image captured by the stereo camera, the distance unit corresponding to the prescribed first distance unit with respect to the traveling direction of the vehicle on the road surface Information for conversion into a first state characteristic value, which is the state characteristic value, and a second state characteristic value, which is the state characteristic value corresponding to a second distance unit having a shorter distance than the first distance unit. Processing method.

Images