JP2023029441A - Measuring device, measuring system, and vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a measuring device, a measuring system, and a vehicle with which it is possible to discriminate a foreign matter that is different from a measurement object and thereby accurately and quickly perform a measurement process in inspecting the measurement object.

SOLUTION: A measuring device comprises: a recognition unit for recognizing, using a discriminator obtained by learning using learning data pertaining to a state of a measurement object, the state of the measurement object in an imaged image having been imaged by an imaging unit; a determination unit for determining a measurement position at which to measure prescribed physical quantity relative to the measurement object on the basis of a result of recognition of the measurement object state by the recognition unit; and a measurement unit for measuring the physical quantity relative to the measurement object at a measurement position determined by the determination unit.

SELECTED DRAWING: Figure 12

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a measuring device, a measuring system and a vehicle.

In July 2014, close visual inspection became stricter in the maintenance and inspection work of infrastructure such as expressways, bridges, and tunnels, and it is necessary to conduct a full inspection by close visual inspection once every five years. For example, when inspecting the road properties of a paved road (hereinafter simply referred to as "road"), typical damage inspection items include cracks, ruts (unevenness in the width direction of the road), and flatness ( It is already known that an inspection is performed using, for example, the degree of unevenness in the traveling direction of the vehicle as an index.

However, current road inspections are performed manually, and there is a risk of human error occurring during inspections. In addition, depending on the inspection item, there are places that must be avoided, such as signs on the road surface, and there is a problem that it takes a lot of time and effort.

As a technology for measuring ruts in such road inspections, a dedicated vehicle is used to record line scan cameras, laser beams, GPS (Global Positioning System) and voice, and the operation is controlled by a PC (Personal Computer). Then, a technique is disclosed in which rutting is measured by a measuring unit (laser beam) and an inspection screen of the measurement position of the measurement information is displayed (for example, Patent Document 1).

However, with the technique described in Patent Document 1, there is a problem that it is not possible to avoid locations where foreign substances such as signs on the road should be avoided when measuring rutting.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and provides a measuring apparatus capable of accurately and quickly performing measurement processing in inspection of an object to be measured by discriminating foreign matter different from the object to be measured. The purpose is to provide systems and vehicles.

In order to solve the above-described problems and achieve the object, the present invention uses a classifier obtained by learning using learning data regarding the state of the object to be measured to determine the state of the object to be measured in a captured image. a determining unit that determines a measurement position for measuring a predetermined physical quantity with respect to the object to be measured based on the recognition result of the state of the object to be measured by the recognizing unit; a measurement unit that measures the physical quantity of the object to be measured at the measurement position, and the measurement unit measures the physical quantity of the object to be measured when the measurement position determined by the determination unit is outside a predetermined range. It is characterized by not measuring physical quantities.

According to the present invention, it is possible to accurately and quickly perform measurement processing in inspection of the object to be measured by discriminating foreign matter different from the object to be measured.

FIG. 1 is a diagram for explaining the measurement of the amount of rutting. FIG. 2 is a diagram for explaining formation of flatness. FIG. 3 is a diagram illustrating an example of the configuration of an imaging system according to the first embodiment; FIG. 4 is a diagram for explaining an image pickup range in the traveling direction of the vehicle by the stereo camera according to the first embodiment. FIG. 5 is a diagram for explaining an imaging range of the vehicle in the road width direction by the stereo camera according to the first embodiment. FIG. 6 is a diagram illustrating overlap in the traveling direction of the vehicle of the stereo imaging ranges of the stereo cameras according to the first embodiment. FIG. 7 is a diagram illustrating an example of an imaging system having three stereo cameras according to the first embodiment. FIG. 8 is a block diagram showing a schematic overall configuration of the imaging system according to the first embodiment. FIG. 9 is a diagram illustrating an example of a schematic hardware configuration of an imaging system according to the first embodiment; 10 is a diagram illustrating an example of a hardware configuration of a stereo camera according to the first embodiment; FIG. 11 is a diagram illustrating an example of a hardware configuration of an information processing apparatus according to the first embodiment; FIG. FIG. 12 is a diagram showing an example of the configuration of functional blocks for explaining functions of the information processing apparatus according to the first embodiment. FIG. 13 is a flowchart illustrating an example of flatness calculation processing according to the first embodiment. FIG. 14 is a flowchart illustrating an example of depth map generation processing according to the first embodiment. FIG. 15 is a diagram for explaining trigonometry. FIG. 16 is a flowchart illustrating an example of camera position and orientation estimation processing according to the first embodiment. FIG. 17 is a diagram for explaining camera position and direction estimation processing according to the first embodiment. FIG. 18 is a flow chart showing an example of a rutting amount measurement process according to the first embodiment. FIG. 19 is a diagram showing an example of an image showing a foreign substance recognition result according to the first embodiment. FIG. 20 is a diagram for explaining the measurement positions of the amount of rutting. FIG. 21 is a diagram showing measurement positions of the amount of rutting. FIG. 22 is a diagram showing an example of the configuration of functional blocks for explaining the functions of the information processing apparatus according to the second embodiment. FIG. 23 is a flow chart showing an example of a rutting amount measurement process according to the second embodiment. FIG. 24 is a diagram showing an example of the configuration of functional blocks for explaining the functions of the information processing apparatus according to the third embodiment. FIG. 25 is a flow chart showing an example of a rutting amount measurement process according to the third embodiment.

EMBODIMENT OF THE INVENTION Below, embodiment of the measuring device which concerns on this invention, a measuring system, and a vehicle is described in detail, referring drawings. In addition, the present invention is not limited by the following embodiments, and components in the following embodiments can be easily conceived by those skilled in the art, substantially the same, and so-called equivalent ranges. are included. Furthermore, various omissions, replacements, changes, and combinations of components can be made without departing from the scope of the following embodiments.

[Overview of existing road property inspection methods]
Before describing the embodiments, an existing method for inspecting road properties will be briefly described. As an index for evaluating road properties, a pavement maintenance control index (MCI: Maintenance Control Index) is established. MCI quantitatively evaluates the usability of pavement using three types of road surface property values: "crack rate", "rutting amount" and "flatness".

Among these, the "crack rate" is obtained by, for example, calculating the crack area according to the number of cracks and the patching area in each area obtained by dividing the road surface into 50 cm meshes, and calculating the calculation result using the following formula. Apply to (1) to find the crack rate.

1 crack ⇒ 0.15 [m ² ] crack 2 cracks ⇒ 0.25 [m ² ] crack Patching area 0 to 25 [%] ⇒ Crack 0 [m ² ]
Patching area 25-75 [%] ⇒ Crack 0.125 [m ² ]
Patching area 75 [%] or more ⇒ Crack 0.25 [m ² ]

As illustrated in FIGS. 1( a ) and 1 ( b ), the “rut amount” is measured by measuring the depths D ₁ and D ₂ of two “ruts” that occur in one lane, for example. The larger value of the depths _D1 and _D2 is adopted. Since there are several patterns in how the ruts are dug, a measurement method suitable for each pattern is adopted. In FIG. 1(a), the part between the two "ruts" is higher than both ends of the two "ruts", and in FIG. 1(b), the part between the two "ruts" is It shows an example of the measurement method when it is lower than both ends of two "ruts".

"Flatness" is measured along the traveling direction of the vehicle on the road surface, for example, by measuring the height from the road surface at three points at intervals of 1.5 [m]. For example, as shown in FIG. 2, measuring instruments are provided at three locations A, B, and C at intervals of 1.5 [m] on the underside of the vehicle to measure heights X ₁ , X _{2 ,} and X ₃ . Based on the measured heights X ₁ , X ₂ and X ₃ , the displacement amount d is obtained by the following formula (2). This measurement is performed multiple times while moving the vehicle. Based on the plurality of displacement amounts d thus obtained, the flatness σ is calculated using the following formula (3).

The above-described "crack rate", "rutting amount" and "flatness" are measured, for example, for each evaluation section of 100 [m]. If the location of the damage is known in advance, the measurement may be performed by dividing 100 [m] into smaller units such as 40 [m]+60 [m]. MCI is calculated based on the results of measurement performed in units of 100 [m], and a record is created.

MCI is calculated using the four formulas shown in Table 1 below based on the measured crack rate C [%], rutting amount D [mm], and flatness σ [mm], and the value MCI , MCI1, MCI2 and MCI3. Then, among the calculated values MCI, MCI1, MCI2 and MCI3, the minimum value is adopted as MCI. Based on the adopted MCI, evaluation is performed according to the evaluation criteria shown in (Table 2) below, and it is determined whether or not the road surface in the evaluation section needs repair.

[First Embodiment]
Next, an imaging system according to the first embodiment will be described. In the first embodiment, the depth information can be obtained by capturing two captured images (hereinafter sometimes referred to as "stereo captured images") by simultaneously capturing images of the object from a plurality of different directions. A stereo camera is attached to the vehicle to capture images of the road surface. Depth information for the road surface is obtained from the imaging position based on the captured stereo images, the three-dimensional shape of the road surface is generated, and three-dimensional road surface data is created. By analyzing this three-dimensional road surface data, it is possible to obtain the "crack rate", "rutting amount" and "flatness" used to obtain the MCI.

More specific description will be given. A stereo camera has two cameras separated by a predetermined length (called a base line length), and outputs a pair of captured images (stereo captured images) captured by the two cameras. By searching for a corresponding point between two captured images included in this stereo captured image, the depth distance can be restored for an arbitrary point in the captured image. Depth distance (hereinafter sometimes referred to as “depth value”) is restored for the entire captured image, and data representing each pixel with depth information is called a depth map. That is, the depth map is three-dimensional point group information consisting of a set of points each having three-dimensional information.

This stereo camera is mounted downward at one position such as the rear of the vehicle so that the ground can be imaged, and the vehicle is moved along the road to be measured. To simplify the explanation, it is assumed that the imaging range of the stereo camera mounted on the vehicle for measurement covers a specified length in the road width direction.

For the rutting amount D, the depth values of the striped portions cut in the road width direction in the depth map restored from the stereo image captured by the stereo camera are arranged. Depths _D1 and _D2 can be calculated based on the road width direction variation of this depth value.

The crack rate C is obtained by analyzing a captured image of the road surface to detect "cracks" and the like, and performing calculation according to Equation (1) described above based on the detection results.

Here, in the traveling direction of the vehicle, since the imaging range is limited in one imaging, for example, the crack rate C in a 100 [m] section cannot be calculated based on the captured image obtained by one imaging. Therefore, while moving the vehicle along the road, images are sequentially captured for each movement corresponding to the length of the imaging range in the direction in which the vehicle travels. At this time, the imaging trigger (imaging trigger) is controlled so that the imaging range of the previous imaging and the imaging range of the current imaging overlap at a predetermined overlapping rate or more.

In this way, by controlling the imaging trigger according to the movement of the vehicle, the road surface to be measured can be imaged without omission. For this reason, captured images that are sequentially captured according to the progress of the vehicle are stitched together using image processing called stitching, for example, to generate a single image including a road surface image of a 100 [m] section. By visually confirming or analyzing this image, the crack rate C on the road surface can be measured.

Flatness measurement uses a technique called Structure from Motion (hereinafter referred to as “SfM”), which estimates the imaging position based on sufficiently overlapping images taken from images at different imaging points.

An outline of the SfM processing will be described. First, using images captured in overlapping imaging ranges, points obtained by imaging the same point in each image are detected as corresponding points. It is desirable to detect as many corresponding points as possible. Next, for example, simultaneous equations are established using the coordinates of the detected corresponding points for the movement of the camera from the imaging point of the first image to the imaging point of the second image, with rotation and translation as unknown parameters. , find the parameters that minimize the total error. In this way, the imaging position of the second image can be calculated.

As described above, the depth of an arbitrary point in the image can be restored as a depth map from the stereo-captured image at each imaging point. The depth map corresponding to the second stereo-captured image is obtained from the simultaneous equations described above with respect to the depth map corresponding to the first stereo-captured image, with the imaging position of the first camera as the origin. Coordinate transformation to map. As a result, the two depth maps can be unified into the coordinate system of the first depth map. In other words, one depth map can be generated by synthesizing two depth maps.

By performing this processing for, for example, depth maps based on all stereo imaged images captured in a 100 [m] section and synthesizing them, the road surface of the 100 [m] section is restored in one three-dimensional space. The amount of displacement d can be calculated by applying the depth value of the road surface thus restored to the above equation (2). Applying this displacement amount d to the above equation (3), the flatness σ is calculated.

In the first embodiment, images are captured by a stereo camera mounted on a vehicle, and image processing is performed on the captured stereo images. , the crack rate C can be collectively measured. In the first embodiment, this measurement can be performed using an imaging system including a stereo camera and an information processing device that performs image processing on stereo captured images, and MCI can be obtained with a simple configuration. Become.

In addition, with the existing technology, each of the three indices for determining MCI is measured by separate devices, so control and management such as data storage and time synchronization between data are complicated. . On the other hand, in the first embodiment, since the three indices for obtaining MCI are calculated based on a common stereo image, control and management such as data storage and time synchronization between data are easy. Become.

(Stereo camera arrangement)
Next, an example of stereo camera arrangement in the first embodiment will be described. FIG. 3 is a diagram illustrating an example of the configuration of an imaging system according to the first embodiment; FIG. 3A is a side view of the vehicle 1 showing how the imaging system according to the first embodiment is mounted on the vehicle 1. FIG. In FIG. 3( a ), the direction toward the left end of the drawing is the traveling direction of the vehicle 1 . That is, in FIG. 3( a ), the left end side of the vehicle 1 is the front portion of the vehicle 1 , and the right end side is the rear portion of the vehicle 1 . FIG. 3(b) is a diagram showing an example of the vehicle 1 viewed from the rear side.

In the imaging system according to the first embodiment, a mounting member 3 having a mounting portion 2 for mounting an imaging device is fixed to the rear portion of a vehicle body 1 as a mobile body on which the imaging system is mounted, and one or more Attach the stereo camera 6 of . Here, as exemplified in FIG. 3B, two stereo cameras 6L and 6R (imaging units) are attached to both end sides of the vehicle body of the vehicle 1 in the width direction. Each of the stereo cameras 6L and 6R is attached in an orientation to image a road surface 4 (an example of an object to be measured) on which the vehicle 1 moves. Preferably, each stereo camera 6L and 6R is mounted so as to image the road surface 4 from the vertical direction.

Hereinafter, the stereo cameras 6L and 6R will be collectively referred to as the stereo camera 6 when there is no need to distinguish between the stereo cameras 6L and 6R.

The stereo camera 6 is controlled by, for example, a personal computer (PC) 5 installed inside the vehicle 1, for example. The operator operates the PC 5 and instructs the stereo camera 6 to start imaging. When instructed to start imaging, the PC 5 starts imaging with the stereo camera 6 . The imaging is controlled in timing according to the moving speed of the stereo camera 6, that is, the vehicle 1, and is repeatedly executed. Note that the device that controls the stereo camera 6 is not limited to the PC 5, and may be a work station, a dedicated device that specializes in controlling the stereo camera 6, or the like.

For example, the operator operates the PC 5 and instructs the end of imaging when the imaging of the required section is completed. The PC 5 terminates the imaging by the stereo camera 6 in response to the imaging termination instruction.

4 and 5 are diagrams showing examples of the imaging range of the stereo camera 6 applicable to the first embodiment. In FIGS. 4 and 5, the same reference numerals are given to the parts common to FIG. 3 described above, and detailed description thereof will be omitted.

FIG. 4 is a diagram for explaining an imaging range (referred to as a traveling direction visual field Vp) in the traveling direction of the vehicle by the stereo camera according to the first embodiment. 4, the direction toward the left end of the drawing is the direction of travel of the vehicle 1, as in FIG. 1 described above. The traveling direction visual field Vp is determined according to the angle of view α of the stereo camera 6 and the height h of the stereo camera 6 with respect to the road surface 4, as shown in FIG.

FIG. 5 is a diagram for explaining an imaging range of the vehicle in the road width direction by the stereo camera according to the first embodiment. FIG. 5 is a view of the vehicle 1 viewed from the rear side, and parts common to those in FIG.

In FIG. 5(a), the stereo camera 6L comprises two imaging lenses 6L- _L and 6L- _R . A line connecting the imaging lenses 6L _L and 6L _R is called a base line, and its length is called a base line length. Similarly, the stereo camera 6R also has two imaging lenses 6R _L and 6R _R separated by the base line length, and is arranged so that the base line is perpendicular to the traveling direction of the vehicle 1 .

FIG. 5(b) shows an example of the imaging range of the stereo cameras 6L and 6R of the first embodiment. In the stereo camera 6L, the imaging ranges 60L _L and 60L _R of the imaging lenses 6L _L and 6L _R are overlapped with a shift according to the baseline length and height h. Similarly for the stereo camera 6R, imaging ranges 60R- _L and 60R- _R respectively by the imaging lenses 6R _-L and 6R- _R are overlapped with a shift according to the base line length and the height h.

Hereinafter, unless otherwise specified, these imaging ranges 60L _L and 60L _R and imaging ranges 60R _L and 60R _R are collectively referred to as stereo imaging ranges 60L and 60R, respectively. As shown in FIG. 5(b), the stereo cameras 6L and 6R have the stereo imaging ranges 60L and 60R, which are a part of the stereo imaging range 60L on one end side in the width direction of the vehicle 1 and the stereo imaging range 60R. are arranged so as to overlap with a part of the region on the other end side in the width direction of the region 61 at a predetermined overlap rate.

FIG. 6 is a diagram showing overlap in the traveling direction of the vehicle 1 of the stereo imaging range 60L by the stereo camera 6L, which is applicable to the first embodiment. In FIG. 6, the left camera field of view V _L and the right camera field of view V _R respectively indicate the fields of view (imaging ranges) of the imaging lenses 6L _L and 6L _R in the stereo camera 6L. Assume that the stereo camera 6L takes two images of the moving vehicle 1 . The stereo camera 6L takes an image of the stereo imaging range 60L (imaging ranges _60LL and _60LR ) for the first time, and moves the vehicle 1 in the traveling direction of the vehicle 1 with respect to the stereo imaging range 60L for the second time. The imaging of the stereo imaging range 60L' (imaging ranges 60L _L ' and 60L _R ') moved by a distance corresponding to the distance is performed.

At this time, the imaging timing of the stereo camera 6L is controlled so that the stereo imaging range 60L and the stereo imaging range 60L' overlap with the traveling direction visual field Vp at an overlap rate equal to or greater than a predetermined travel direction overlap rate Dr. do. In this way, by sequentially imaging the stereo imaging ranges 60L and 60L' along the time axis, it is possible to facilitate the process of joining two stereo imaging images obtained by imaging the stereo imaging ranges 60L and 60L'. .

Although the imaging system according to the first embodiment uses the two stereo cameras 6L and 6R in the above description, the imaging system is not limited to this example. For example, as shown in FIG. 7(a), the imaging system according to the first embodiment further adds one stereo camera 6C to the stereo cameras 6L and 6R, resulting in three stereo cameras 6L and 6R. and 6C.

In the example of FIG. 7(a), the distance between the stereo cameras 6L and 6R is wider than when only the stereo cameras 6L and 6R are used, and the stereo camera 6C is arranged in the center. As shown in FIG. 7B, imaging ranges 60C _L and 60C _R formed by imaging lenses 6C _L and 6C _R of the stereo camera 6C constitute a stereo imaging range 60C. Stereo cameras 6L, 6C and 6R are arranged such that respective stereo imaging ranges 60L, 60C and 60R overlap in the width direction of vehicle 1 at a predetermined overlap rate.

In this way, by using the three stereo cameras 6L, 6C and 6R to capture an image of one lane, it is possible to capture images by setting the image capturing ranges on the right side, the center and the left side of the lane, respectively, resulting in high image quality ( (high resolution) can be captured by a small number of stereo cameras. Here, especially the road width is generally defined as 3.5 [m]. Therefore, it is conceivable to image both ends of the lane in the road width direction with the stereo cameras 6L and 6R and image the central portion with the stereo camera 6C, corresponding to the road width of 3.5 [m].

Note that the imaging system may be configured using only one stereo camera having an angle of view that allows the imaging range to include the defined road width (3.5 [m]).

(Configuration of imaging system)
Next, the configuration of the imaging system according to the first embodiment will be described. In the following description, it is assumed that the imaging system has two stereo cameras.

FIG. 8 is a block diagram showing an example of a schematic configuration of an imaging system according to the first embodiment; In FIG. 8, the imaging system 10 includes imaging units _100-1 and _100-2 , imaging control units _101-1 and _101-2 , a speed acquisition unit 102, and a generation unit 103.

The imaging units _100-1 and _100-2 correspond to the stereo cameras 6L and 6R described above, respectively. The imaging control units _101-1 and _101-2 control imaging operations such as imaging timing, exposure, and shutter speed of the imaging units _100-1 and _100-2 , respectively. The velocity acquisition unit 102 acquires the velocity of the imaging units _100-1 and _100-2 with respect to the subject (road surface 4). The generation unit 103 generates a trigger (imaging trigger) for designating imaging by the imaging units 100 ₁ and 100 ₂ based on the speed acquired by the speed acquisition unit 102 and the traveling direction visual field Vp. The generation unit 103 sends the generated triggers to the imaging control units _101-1 and _101-2 . The imaging control units _101-1 and _101-2 cause the imaging units _100-1 and _100-2 to perform imaging operations in response to the trigger sent from the generation unit 103. FIG.

FIG. 9 is a diagram illustrating an example of a schematic hardware configuration of an imaging system according to the first embodiment; In FIG. 9, the imaging system 10 includes stereo cameras 6L and 6R, and an information processing device 50 (an example of a measuring device) corresponding to the PC 5 in FIG. Information processing device 50 generates a trigger at a predetermined timing and sends the generated trigger to stereo cameras 6L and 6R. The stereo cameras 6L and 6R take images in response to this trigger. Each stereo captured image captured by stereo cameras 6L and 6R is transmitted to information processing device 50 . The information processing device 50 stores and accumulates the stereo captured images received from the stereo cameras 6L and 6R in a storage or the like. The information processing device 50 performs image processing such as generation of a depth map and joining of the generated depth maps based on the accumulated stereo captured images.

FIG. 10 is a diagram showing an example of the hardware configuration of the stereo camera 6L according to the first embodiment. Note that the stereo camera 6R can be realized with a configuration similar to that of the stereo camera 6L.

10, the stereo camera 6L includes imaging optical systems 600 _L and 600 _R , imaging elements 601 _L and 601 _R , driving units 602 _L and 602 _R , signal processing units 603 _L and 603 _R , and an output unit 604 and including. Among these, the imaging optical system _600L has a configuration corresponding to the imaging lens _6LL described above. Similarly, the imaging optical system 600 _R has a configuration corresponding to the imaging lens 6L _R described above.

The imaging optical system _600L is an optical system having an angle of view α and a focal length f, and projects light from a subject onto the imaging element _601L . The imaging element 601 _L is, for example, an optical sensor using a CMOS (Complementary Metal Oxide Semiconductor), and outputs a signal corresponding to projected light. Note that an optical sensor using a CCD (Charge Coupled Device) may be applied to the imaging element _601L . The drive unit _602L drives the image sensor _601L , performs predetermined processing such as noise removal and gain adjustment on the signal output from the image sensor _601L , and outputs the processed signal. The signal processing unit 603 _L performs A/D conversion on the signal output from the driving unit 602 _L to convert the signal into a digital image signal (captured image). The signal processing unit _603L performs predetermined image processing such as gamma correction on the converted image signal and outputs the processed image signal. The captured image output from the signal processing unit 603L is sent to the output unit 604. FIG.

The operations of the imaging optical system 600 _R , the imaging device 601 _R , the driving unit 602 _R , and the signal processing unit 603 _R are similar to those of the imaging optical system 600 _L , the imaging device 601 _L , the driving unit 602 _L , and the signal processing unit described above. Similar to _603L .

Triggers output from the information processing device 50, for example, are transmitted to the driving units _602L and _602R . The driving units 602 _L and 602 _R take in signals from the imaging elements 601 _L and 601 _R at the timing of the received triggers and perform imaging.

Here, the drive units 602 _L and 602 _R perform exposure in the imaging elements 601 _L and 601 _R by a batch simultaneous exposure method. This method is called a global shutter. On the other hand, since the rolling shutter takes in light in order from the top of the pixel position (line order), each line in the frame does not exactly capture the subject at the same time. In the case of the rolling shutter method, if the camera or the subject moves at high speed while the imaging signal of one frame is captured, the image of the subject is captured shifted according to the line position. Therefore, in the stereo cameras 6L and 6R according to the first embodiment, a global shutter is used so that the road shape is captured correctly in terms of projection geometry.

The output unit 604 outputs the captured images of each frame supplied from the signal processing units _603L and _603R as a set of stereo captured images. The stereo captured images output from the output unit 604 are sent to the information processing device 50 and accumulated.

11 is a diagram illustrating an example of a hardware configuration of an information processing apparatus according to the first embodiment; FIG. 11, the information processing device 50 includes a CPU (Central Processing Unit) 5000, a ROM (Read Only Memory) 5001, a RAM (Random Access Memory) 5002, and a graphics I/F (interface) connected to a bus 5030. ) 5003 , a storage 5004 , an input device 5005 , a data I/F 5006 and a communication I/F 5007 . Furthermore, the information processing apparatus 50 includes a camera I/F 5010 and a speed acquisition section 5021, which are connected to the bus 5030, respectively.

The storage 5004 is a storage medium that stores data in a nonvolatile manner, and can be a hard disk drive or flash memory. Storage 5004 stores programs and data for CPU 5000 to operate.

The CPU 5000 controls the overall operation of the information processing apparatus 50 using the RAM 5002 as a work memory according to a program pre-stored in at least one of the ROM 5001 and the storage 5004, for example. The graphics I/F 5003 generates display signals that can be handled by the display 5020 based on display control signals generated by the CPU 5000 according to a program. A display 5020 displays a screen according to the display signal supplied from the graphics I/F 5003 .

The input device 5005 receives user operations and outputs control signals according to the received user operations. As the input device 5005, a pointing device such as a mouse or tablet, a keyboard, or the like can be applied. Further, the input device 5005 and the display 5020 may be integrally formed to form a so-called touch panel configuration.

A data I/F 5006 transmits and receives data to and from an external device. As the data I/F 5006, for example, USB (Universal Serial Bus) is applicable. A communication I/F 5007 controls communication with an external network according to instructions from the CPU 5000 .

A camera I/F 5010 is an interface for each of the stereo cameras 6L and 6R. Each stereo picked-up image output from each stereo camera 6L and 6R is passed to CPU5000 via camera I/F5010. In addition, the camera I/F 5010 generates the above-described triggers according to instructions from the CPU 5000, and sends the generated triggers to the stereo cameras 6L and 6R.

The speed acquisition unit 5021 acquires speed information indicating the speed of the vehicle 1 . When the stereo cameras 6L and 6R are attached to the vehicle 1, the speed information obtained by the speed obtaining unit 5021 indicates the speed of the stereo cameras 6L and 6R with respect to the subject (road surface). The speed acquisition unit 5021 has, for example, a function of receiving a GNSS (Global Navigation Satellite System) signal, and acquires speed information indicating the speed of the vehicle 1 based on the Doppler effect of the received GNSS signal. The speed acquisition unit 5021 can also acquire the speed information directly from the vehicle 1 without being limited to this.

FIG. 12 is a diagram showing an example of the configuration of functional blocks for explaining functions of the information processing apparatus according to the first embodiment. In FIG. 12 , the information processing apparatus 50 includes a captured image acquisition section 500 , a UI section 501 , a control section 502 and an imaging control section 503 . The information processing device 50 further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, and a third It includes a state characteristic value calculation unit 523 , a record creation unit 524 , a storage unit 530 and a notification unit 531 .

These captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic Value calculator 522 , third state characteristic value calculator 523 , and report preparation unit 524 are implemented by programs running on CPU 5000 . The captured image acquisition unit 500, the UI unit 501, the control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, and the first state characteristic value calculation unit 521 are not limited to these. , the second state characteristic value calculation unit 522, the third state characteristic value calculation unit 523, and the report creation unit 524 may be partially or entirely configured by hardware circuits that operate in cooperation with each other.

The captured image acquisition unit 500 acquires stereo captured images from the stereo cameras 6L and 6R. The captured image acquisition unit 500 stores the acquired stereo captured images in the storage 5004, for example. Also, the captured image acquisition unit 500 acquires the stored stereo captured image from the storage 5004, for example.

The UI unit 501 implements a user interface by displaying on the input device 5005 and display 5020 . A control unit 502 controls the operation of the information processing apparatus 50 as a whole.

The imaging control unit 503 corresponds to the imaging control units 101 ₁ and 101 ₂ , the velocity acquisition unit 102 and the generation unit 103 described above. That is, the imaging control unit 503 acquires speed information indicating the speed of each of the stereo cameras 6L and 6R with respect to the subject (road surface 4), and combines the acquired speed information with the preset angle of view α of each of the stereo cameras 6L and 6R. and height h, a trigger is generated for instructing the imaging of each of the stereo cameras 6L and 6R.

The matching processing unit 510 performs matching processing using the two captured images forming the stereo captured image acquired by the captured image acquisition unit 500 . The 3D information generation unit 511 performs processing related to 3D information. For example, the 3D information generation unit 511 obtains depth information by trigonometry or the like using the result of matching processing by the matching processing unit 510, and generates 3D point cloud information (depth map) based on the obtained depth information.

The 3D information acquisition unit 520 acquires the 3D point cloud information obtained for each stereo image by the 3D information generation unit 511 .

The first state characteristic value calculation unit 521 uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to determine the state for obtaining MCI. Among the characteristic values, the crack rate C is calculated.

The second state characteristic value calculation unit 522 uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to determine the state for obtaining MCI. Among the characteristic values, flatness σ is calculated.

The third state characteristic value calculation unit 523 uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to determine the state for obtaining MCI. Among the characteristic values, a rutting amount D (an example of a physical quantity) is calculated. The third state characteristic value calculation unit 523 has an input unit 5231, a recognition unit 5232, a determination unit 5233, a measurement unit 5234, and an output unit 5235, as shown in FIG.

The input unit 5231 inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 . The stereo captured image input by the input unit 5231 may be any of the stereo captured images captured by the stereo cameras 6L and 6R.

The recognition unit 5232 uses a discriminator, which is a learning model obtained by supervised learning (deep learning or machine learning) based on learning data about the foreign matter information on the road surface added in advance. To recognize whether or not a foreign object is included in a subject indicated by a stereo image. At this time, the recognizing unit 5232 reads data of the classifier from the storage unit 530 .

The determining unit 5233 determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point group information (depth map) based on the recognition result of the foreign matter in the stereo imaged image by the recognizing unit 5232 .

The measurement unit 5234 measures the rutting amount D at the measurement position in the three-dimensional point group information determined by the determination unit 5233 .

The output unit 5235 outputs the rutting amount D measured by the measurement unit 5234 to the report creation unit 524 .

The record creation unit 524 obtains MCI based on each state characteristic value calculated by the first state characteristic value calculation unit 521, the second state characteristic value calculation unit 522, and the third state characteristic value calculation unit 523, and creates a record. .

The storage unit 530 stores, for example, classifier data that is a learning model used by the recognition unit 5232 described above. Storage unit 530 is realized by RAM 5002 or storage 5004 shown in FIG.

Notification unit 531 displays and notifies various information on display 5020 . The notification unit 531 notifies, for example, that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to the notification by the display on the display 5020, and may be notified by, for example, sound or electronic sound such as an alarm sound.

A program for realizing each function in the information processing apparatus 50 according to the first embodiment is stored as an installable or executable file on a CD (Compact Disk), a flexible disk (FD), or a DVD (Digital Versatile). Disk) or other computer-readable recording medium. Alternatively, the program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. Also, the program may be configured to be provided or distributed via a network such as the Internet. It should be noted that the provision or distribution of the above-described program is the same in the second and third embodiments described later.

The program includes a captured image acquisition unit 500, a UI unit 501, a control unit 502, an imaging control unit 503, a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a It has a module configuration including a 2-state characteristic value calculator 522 , a 3rd state characteristic value calculator 523 , a report preparation unit 524 and a notification unit 531 . As actual hardware, the CPU 5000 reads out the program from a storage medium such as the storage 5004 and executes it, thereby loading the above-described respective units onto the main storage device such as the RAM 5002 . , control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value A calculation unit 523, a report creation unit 524, and a notification unit 531 are generated on the main storage device. The configuration of the above program and the mode of generation on the main storage device are the same for the second and third embodiments described later.

(Trigger generation method)
Next, a method of generating triggers for instructing imaging to the stereo cameras 6L and 6R according to the first embodiment will be described in more detail. In the first embodiment, the generation unit 103 calculates the distance in the direction of the speed of the vehicle 1 in the stereo imaging range of the object to be measured (road surface 4) of the stereo cameras 6L and 6R at the speed indicated by the speed information. Triggers are generated at time intervals equal to or less than a predetermined time shorter than the moving time.

That is, the trigger needs to be generated so that the photographing ranges of the road surface 4 captured by the stereo cameras 6L and 6R overlap in the traveling direction of the vehicle 1 at a predetermined overlapping rate (traveling direction overlapping rate Dr) or more. be. As will be described later, the purpose of this is to detect sufficient corresponding points in order to stably and accurately calculate the camera position in the process of calculating the imaging position from each stereo imaged image. The lower limit of the traveling direction overlap rate Dr is experimentally determined, for example, as "60[%]".

In the first embodiment, the following three methods can be applied to the trigger generation method.
(1) Method of generating at regular time intervals (first generation method)
(2) A method of generating by detecting the movement speed of the camera (second generation method)
(3) A method of calculating and generating a movement distance using a captured image (third generation method)

<First generation method>
First, the first trigger generation method will be described. In the first generation method, the trigger time interval is determined from the maximum speed Speed of the vehicle 1 during imaging and the size of the imaging range (the length of the imaging range in the traveling direction of the vehicle 1). The speed acquisition unit 5021 acquires the maximum speed Speed of the vehicle 1 in advance based on system setting values in the vehicle 1 and user input to the information processing device 50 . The imaging control unit 503 acquires the maximum speed Speed from the speed acquisition unit 5021, and uses the acquired maximum speed Speed, the traveling direction visual field Vp, and the traveling direction overlap rate Dr to obtain the trigger time interval by the following formula: Calculated by (4). Note that the above-described lower limit value is applied to the traveling direction overlapping rate Dr.

The number of triggers fps to be generated per second is calculated by the formula (4). The reciprocal of the number of triggers fps is the time interval until the next trigger to be generated.

As described with reference to FIG. 4, the traveling direction field of view Vp is schematically defined by the height h of each of the stereo cameras 6L and 6R from the road surface 4 and the angle of view α of each of the stereo cameras 6L and 6R. can be set based on In practice, furthermore, the angles of the stereo cameras 6L and 6R with respect to the road surface 4 are taken into consideration when setting the traveling direction field of view Vp.

Here, when the moving vehicle 1 curves to the right or left, the amount of movement of the outer camera out of the stereo cameras 6L and 6R increases. Therefore, it is more preferable to use the speed of rotational movement based on the position of the outer camera instead of using the maximum speed Speed of the vehicle 1 as it is.

The imaging control unit 503 stores and accumulates all the stereo imaged images imaged according to the generated trigger in the storage 5004 or the RAM 5002, for example.

<Second generation method>
Next, a second trigger generation method will be described. While the above-described first method is simple, when the vehicle 1 is stopped or moving at a speed lower than a predetermined speed, images are captured at excessively fine intervals with respect to the maximum speed Speed. As a result, the amount of stereo imaged images to be accumulated becomes large. In the second generation method, the imaging control unit 503 detects the moving speed of the camera and generates a trigger according to the detected moving speed.

The image capture control unit 503 uses the current speed of the vehicle 1 indicated by the speed information acquired by the speed acquisition unit 5021 as the maximum speed Speed in expression (4) to calculate the time interval until the next trigger, and captures the image. I do. According to the second generation method, the slower the moving speed of the vehicle 1 is, the longer the time interval of trigger generation is, thereby suppressing wasteful imaging.

The imaging control unit 503 stores and accumulates all the stereo imaged images imaged according to the generated trigger in the storage 5004 or the RAM 5002, for example.

Note that this second generation method and the above-described first generation method can be implemented in combination.

<Third generation method>
Next, a third trigger generation method will be described. In the third generation method, as in the first generation method described above, triggers are generated at regular time intervals based on the maximum speed Speed of the vehicle 1 . Here, in the third generation method, instead of accumulating all the stereo captured images captured in response to the trigger, the last accumulated stereo captured image and the immediately preceding stereo captured image are combined. Accumulation is performed only when the overlapping rate is less than the preset traveling direction overlapping rate Dr.

As will be described later, it is possible to calculate the moving distance of the camera (vehicle 1) using only the stereo captured images captured so as to overlap with the traveling direction of the vehicle 1 .

The imaging control unit 503 calculates the moving distance of the camera using the last accumulated stereo image and the immediately preceding stereo image. The imaging control unit 503 determines whether or not the calculated distance exceeds the movement distance corresponding to the lower limit of the traveling direction overlapping rate Dr. When the image capturing control unit 503 determines that the time limit has been exceeded, the image capturing control unit 503 accumulates the stereo image captured immediately before. On the other hand, if it is determined that the imaging control unit 503 does not exceed the threshold, the imaging control unit 503 discards the stereo image captured immediately before.

As a result, even without using a sensor for measuring the current speed of the vehicle 1, wasteful image accumulation is not performed when the moving speed is low.

In the above description, it is assumed that triggers for instructing imaging to the stereo cameras 6L and 6R are generated outside the stereo cameras 6L and 6R, for example, in the information processing device 50, and transmitted to the stereo cameras 6L and 6R. Yes, but not limited to. For example, the trigger may be generated by one of the stereo cameras 6L and 6R to instruct the stereo cameras 6L and 6R to take an image.

(Calculation method of road surface property value)
Next, a method for calculating road surface property values according to the first embodiment will be described. A method for measuring the flatness σ, the crack ratio C, and the rutting amount D according to the first embodiment will be described below.

<Flatness>
FIG. 13 is a flowchart illustrating an example of flatness calculation processing according to the first embodiment. In step S100, the captured image acquisition unit 500 acquires stereo captured images captured by the stereo cameras 6L and 6R and stored in the storage 5004, for example. When the stereo imaged images are acquired, the process moves to steps S101a and S101b, which can be processed in parallel.

In step S101a, a depth map is generated based on the stereo captured image acquired in step S100. Depth map generation processing applicable to the first embodiment will be described with reference to FIGS. 14 and 15 . FIG. 14 is a flowchart illustrating an example of depth map generation processing according to the first embodiment.

In step S<b>120 , the matching processing unit 510 acquires stereo captured images from the captured image acquisition unit 500 . In the next step S121, the matching processing unit 510 performs matching processing based on the two captured images forming the acquired stereo captured image. In the next step S122, the 3D information generation unit 511 calculates depth information based on the matching processing result of step S121, and generates a depth map, which is three-dimensional point cloud information.

The processing of steps S121 and S122 will be described more specifically. In the first embodiment, depth information is calculated by a stereo method using two captured images that constitute a stereo captured image. The stereo method here uses two captured images captured from different viewpoints by two cameras, and for a certain pixel (reference pixel) in one captured image, a corresponding pixel (reference pixel) in the other captured image ( In this method, the corresponding pixel) is obtained, and the depth (depth value) is calculated by triangulation based on the reference pixel and the corresponding pixel.

In step S121, the matching processing unit 510 uses the two captured images constituting the stereo captured image, which are acquired from the captured image acquisition unit 500, and a predetermined size centering on the reference pixel in one captured image serving as a reference. A search is performed by moving a region corresponding to the region in the other captured image to be searched within the other captured image.

Various methods are known for searching for corresponding pixels, and for example, a block matching method can be applied. The block matching method acquires pixel values of a region cut out as a block of M pixels×N pixels centering on a reference pixel in one captured image. Also, in the other captured image, a pixel value of a region cut out as a block of M pixels×N pixels centering on the target pixel is acquired. Based on the pixel values, the degree of similarity between the area containing the reference pixel and the area containing the target pixel is calculated. The similarity is compared while moving blocks of M pixels×N pixels in the search target image, and the target pixel in the block at the position where the similarity is highest is taken as the corresponding pixel corresponding to the reference pixel.

Similarity can be calculated by various calculation methods. For example, the normalized cross-correlation (NCC) shown in the following formula (5) is one of the cost functions, and the larger the value of the numerical value C _NCC indicating the cost, the higher the similarity. Indicates high. In equation (5), the values M and N represent the size of the pixel block to search. Also, the value I(i, j) represents the pixel value of a pixel in a pixel block in one of the reference captured images, and the value T(i, j) represents the pixel block in the other captured image to be searched. represents the pixel value in

As described above, the matching processing unit 510 moves a pixel block in the other captured image corresponding to a pixel block of M pixels×N pixels in the other captured image, for example, in units of pixels. Calculation of equation (5) is performed while allowing the value _{C_NCC} to be calculated. In the other captured image, the center pixel of the pixel block with the largest numerical value C _NCC is set as the target pixel corresponding to the reference pixel.

Returning to the description of FIG. 14, in step S122, the 3D information generation unit 511 calculates a depth value (depth information) using trigonometry based on the reference pixels and corresponding pixels obtained by the matching processing in step S121. Then, three-dimensional point cloud information relating to one captured image and the other captured image that constitute the stereo captured image is generated.

FIG. 15 is a diagram for explaining trigonometry. Calculating the distance S to the target object 403 (for example, one point on the road surface 4) in the figure from the imaging position information in the image captured by each imaging element 402 (corresponding to the imaging elements 601 _L and 601 _R ). is the purpose of processing. That is, this distance S corresponds to the depth information of the target pixel. The distance S is calculated by Equation (6) below.

In equation (6), the value baseline represents the length of the baseline (baseline length) between the cameras 400a and 400b. In the example of FIG. 5, this corresponds to the baseline length by the imaging lenses 6L _L and 6L _R (in the case of the stereo camera 6L). The value f represents the focal length of the lens 401 (corresponding to the imaging lenses 6L _L and 6L _R ). The value q represents parallax. The parallax q is a value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel by the pixel pitch of the imaging element. The coordinate values of the corresponding pixels are obtained based on the result of matching processing in step S121.

This formula (6) is a method of calculating the distance S when the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R are used. This is to calculate the distance S from the captured images captured by the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R , respectively. In the first embodiment, the calculation method based on this equation (6) is applied to each captured image captured by the imaging lenses 6L _L and 6L _R of the stereo camera 6L and the imaging lenses 6R _L and 6R _R of the stereo camera 6R. applied to calculate the distance S for each pixel.

Returning to the description of FIG. 13, in step S101b, the 3D information generation unit 511 estimates the camera position and orientation. Here, the camera position indicates, for example, the coordinates of the center of the stereo camera 6L when viewed as one unit. Details of the processing in step S101b will be described later.

When the processing of steps S101a and S101b ends, the processing moves to step S102. In step S102, the 3D information generating unit 511 converts two depth maps that are adjacent in time series to match the coordinate system of the depth map at the previous time based on the camera position and orientation estimated in step S101b. Coordinate transformation of the depth map of the next time after the time is performed.

Note that the previous time and the next time are the first imaging and the second imaging that are continuously executed in chronological order in response to a trigger in the stereo cameras 6L and 6R. The time is the previous time, and the time when the second imaging is performed is the next time. That is, the depth map at the previous time is a depth map generated based on the stereo image captured by the first imaging, and the depth map at the next time is a depth map generated based on the stereo image captured by the second imaging. be.

In the next step S103, the 3D information generation unit 511 integrates the depth map of the next time coordinate-transformed in step S102 into the depth map of the previous time. That is, the two depth maps of the previous time and the next time are arranged in a common coordinate system by the coordinate conversion in step S102, so that these two depth maps can be integrated. The processes of steps S102 and S103 are performed on stereo captured images captured at all times.

As shown in FIGS. 5 and 7, when the stereo cameras 6L and 6R or the stereo cameras 6L, 6C and 6R are arranged side by side in the road width direction, the stereo cameras arranged in the road width direction (for example, the stereo cameras In the cameras 6L and 6R) as well, the camera position and orientation are similarly estimated, and depth map integration is then performed.

Further, the moving distance of the camera (vehicle 1) in the above-described third generation method can be calculated in the coordinate conversion in step S102 and the depth map integration processing in step S103.

In the next step S104, the 3D information acquisition unit 520 acquires the depth map integrated in step S103 from the 3D information generation unit 511. FIG. Then, the second state characteristic value calculator 522 calculates the flatness σ based on the depth map acquired by the 3D information acquirer 520 . That is, when the depth maps are integrated, the road surface shape of the road in the measured section is restored as a point group in one three-dimensional space. When the point cloud is restored, for each sampling point on the road surface, take a coordinate system that connects the coordinates of the 1.5m points before and after according to the regulations, and calculate the distance to the 3D point cloud at the center, using the formula The amount of displacement d can be calculated in the same manner as in (2), and the flatness σ can be calculated from the amount of displacement d according to equation (3).

Further, regarding the rutting amount D, in step S104, the third state characteristic value calculation unit 523 scans the depth map acquired by the 3D information acquisition unit 520 in the road width direction, Depths _D1 and _D2 as shown in FIG. 1(b) can be obtained. The rutting amount D can be obtained based on the obtained depths _D1 and _D2 and the cross-sectional information obtained by scanning the depth map. The details of the measurement of the rutting amount D will be described later.

<Crack rate>
The first state characteristic value calculator 521, for example, applies the stereo captured image acquired in step S100 to the depth map integrated in step S103. That is, the first state characteristic value calculation unit 521 integrates each stereo captured image that is redundantly captured in the direction of travel of the vehicle 1 with an overlap rate equal to or greater than the travel direction overlap rate Dr. The first state characteristic value calculation unit 521 sets a mesh of 50 [cm] according to regulations for the integrated image, acquires information such as cracks and patching in each mesh by image analysis, and uses the acquired information Calculate the crack rate C according to the regulations.

<Processing for estimating camera position and orientation>
Next, the estimation processing of the camera position and direction in step S101b in the flowchart of FIG. 13 described above will be described in more detail. FIG. 16 is a flowchart illustrating an example of camera position and orientation estimation processing according to the first embodiment. In a first embodiment, the camera position and orientation are estimated using the SfM described above.

In step S<b>130 , the 3D information generation unit 511 acquires stereo captured images from the captured image acquisition unit 500 . Here, the 3D information generation unit 511 acquires two stereo captured images (a stereo captured image at the previous time and a stereo captured image at the next time) continuously captured in time series. Next, in step S131, the 3D information generation unit 511 extracts feature points from each acquired stereo image. This process of extracting feature points is a process of finding points that can be easily detected as corresponding points in each stereoscopic image. to detect

In the next step S132, the 3D information generation unit 511 detects, from the stereo image of the next time, the point captured at the same place as the point extracted as the feature point in the stereo image of the previous time. For this detection processing, a technique called optical flow or a technique called feature point matching represented by SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), and the like can be applied.

At the next step S133, the 3D information generation unit 511 estimates the initial position and orientation of the camera. The 3D information generating unit 511 fixes the coordinates of the corresponding points detected in each stereoscopic image in step S132, and solves the simultaneous equations using the position and orientation of the camera that captured the stereoscopic image at the next time as parameters. , the position of the camera is estimated, and the three-dimensional coordinates are calculated (step S134).

A method for estimating the initial position and orientation of the camera will be described with reference to FIG. Equation (7) below expresses the relationship between the coordinates x _ij of the space point X _j projected onto the camera (viewpoint) P _i . In equation (7), the value n represents the number of points in the three-dimensional space, and the value m represents the number of cameras (captured images).

Each value P is a projection matrix that transforms the coordinates of a point in the three-dimensional space into the two-dimensional coordinates of the image for each camera, and is represented by the following equation (8). Equation (8) consists of a 2-row, 2-column transformation matrix of the three-dimensional coordinates shown on the right side, and a projective transformation f _i from the three-dimensional coordinates to the two-dimensional coordinates.

In FIG. 17, the value X _j and the value P _i are calculated using simultaneous equations given only the coordinates x _ij captured by the camera. A linear least-squares method can be used to solve this simultaneous equation.

In the next step S135, the 3D information generation unit 511 optimizes the three-dimensional coordinates indicating the camera position calculated in step S134. The position and orientation of the camera calculated by solving the system of equations may not have sufficient accuracy. Therefore, the accuracy is improved by performing the optimization process using the calculated value as the initial value.

Two-dimensional coordinates x _ij (reprojection (called coordinates) and the difference is called the residual. For all corresponding points in all stereoscopic images used to calculate the position and orientation of the camera, the parameters are adjusted by an optimization operation so that the sum of residual errors is minimized. This is called bundle adjustment. Global optimization by bundle adjustment can improve the accuracy of camera position and orientation.

The above is the base processing of camera position and orientation estimation in normal SfM. In the first embodiment, the stereo cameras 6L and 6R with known base lengths are used. Therefore, it is easy to search for corresponding points in two captured images that constitute a stereo captured image. Therefore, the three-dimensional coordinate Xj in the space is fixed on the actual scale (actual size), and the position and orientation of the camera after movement can be stably calculated.

<Amount of rutting>
Next, the process of measuring the amount of rutting D in step S104 in the flowchart of FIG. 13 described above will be described in more detail. FIG. 18 is a flow chart showing an example of a rutting amount measurement process according to the first embodiment. FIG. 19 is a diagram showing an example of an image showing a foreign substance recognition result according to the first embodiment. FIG. 20 is a diagram for explaining the measurement positions of the amount of rutting. FIG. 21 is a diagram showing measurement positions of the amount of rutting.

In step S200, the input unit 5231 of the third state characteristic value calculation unit 523 receives the depth map (three-dimensional point group information) acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. An image (hereinafter simply referred to as a captured image in FIG. 18) is input.

In the next step S201, the recognizing unit 5232 of the third state characteristic value calculating unit 523 uses a discriminator, which is a learning model obtained by supervised learning based on learning data about foreign matter information on the road surface added in advance. , recognizes whether or not the object indicated by the captured image input by the input unit 5231 includes a foreign object. At this time, the recognizing unit 5232 reads data of the classifier from the storage unit 530 .

In the next step S202, the recognition unit 5232 determines whether or not the recognition processing using the classifier has been completed for all the captured images input by the input unit 5231. FIG. If the final image has been captured (step S202: Yes), the process proceeds to step S203, and if not (step S202: No), the process returns to step S200.

In the next step S<b>203 , the recognition unit 5232 outputs the recognition result of the captured image using the classifier to the determination unit 5233 . Here, FIG. 19 shows the recognition result of a foreign substance in the captured image using the discriminator by the recognition unit 5232 . The identification image CI showing the recognition result for the captured image in FIG. 901, a white line portion 902, a road sign portion 903, a road shoulder portion 904, and others 905 are recognized as foreign substances. Conversely, in the identification image CI, regions other than these regions recognized as foreign matter are recognized as having no foreign matter.

In the next step S204, the determination unit 5233 of the third state characteristic value calculation unit 523 determines an initial measurement position for measuring the amount of rutting in the captured image input by the input unit 5231. FIG. Then, the determining unit 5233 determines a measurement section (predetermined range) surrounded by a predetermined length in each of the vehicle width direction and the traveling direction of the vehicle 1, with each determined initial measurement position as the center in the traveling direction of the vehicle 1. set.

The determination unit 5233 determines initial measurement positions at appropriate intervals, for example, five locations within a range of 100 [m] in the traveling direction. For example, as shown in FIG. 20 , the determination unit 5233 determines 10 [m], 30 [m], 50 [m], 70 [m], and 90 [m] within a range of 100 [m] in the traveling direction. ] are determined as initial measurement positions. Then, as shown in FIG. 20, the determination unit 5233 determines the predetermined measurement width defined by the measurement range line in the vehicle width direction of the vehicle 1 and the determined initial measurement positions in the traveling direction of the vehicle 1. A measurement section having a length of 10 [m] and 5 [m] in front and rear is set as the center. In FIG. 20, measurement sections A to E correspond to initial measurement positions of 10 [m], 30 [m], 50 [m], 70 [m], and 90 [m], respectively. In FIG. 20, the direction from the measured section A to the measured section E is the traveling direction of the vehicle 1, and the same applies to FIG. 21 described later.

In the next step S205, the determination unit 5233 determines whether or not a foreign object is avoided at a position in the image (identification image) indicating the recognition result corresponding to the determined measurement position in the captured image. If the foreign matter is avoided (step S205: Yes), the process proceeds to step S207, and if the foreign matter is not avoided (step S205: No), the process proceeds to step S206. If the foreign object is not avoided, the notification unit 531 may notify that the rutting amount D cannot be measured at the measurement position.

In step S206, if a foreign object exists at a position in the image (identification image) indicating the recognition result corresponding to the determined measurement position in the captured image, the determination unit 5233 sets the measurement position to the front side in the traveling direction of the vehicle 1, Alternatively, it is shifted backward by a predetermined length (for example, by a specific pixel) and determined as a new measurement position (that is, the measurement position is changed). For example, as shown in FIG. 21( a ), when a foreign object exists at the initial measurement position determined in the measurement section, the determination unit 5233 determines the front side of the traveling direction of the vehicle 1 as shown in FIG. 21( b ). (Left) is shifted by a predetermined length and determined as a new measurement position. And FIG.21(b) has shown that the foreign material was able to be avoided by the new measurement position as a result of shifting the measurement position. After determining the new measurement position, the process returns to step S205.

Here, as a method of shifting the measurement position, for example, a method of continuously shifting a predetermined length forward in the traveling direction of the vehicle 1 to a position where the foreign object is avoided, or a method of continuing to shift the position to the rear in the traveling direction of the vehicle 1 to a position where the foreign object is avoided. There is a method of continuing to shift by a predetermined length up to, or a method of continuing to shift by a predetermined length alternately forward and backward in the direction of travel, but either method may be adopted. Of these methods, in the case of the method of continuing to shift by a predetermined length alternately forward and backward in the traveling direction, the final measurement position can be determined as close to the initial position as possible, and the relation with other measurement sections can be determined respectively. It is possible to suppress variations in the determined measurement positions, and accurate measurement within a 100 [m] section is possible. It should be noted that the method of shifting the measurement position is the same in the second and third embodiments, which will be described later.

In step S206, if the new measurement position determined by the determination unit 5233 is out of the measurement section, the measurement unit 5234 may not measure the rutting amount D at the measurement position. In this case, the notification unit 531 may notify that the rutting amount D is not measured at the measurement position.

In step S207, if there is no foreign object at the position in the image (identification image) indicating the recognition result corresponding to the measurement position determined in the captured image, the determination unit 5233 measures the rutting amount D at the measurement position. Determined as the measurement position. The determination unit 5233 thus determines the measurement position for measuring the rutting amount D in each of the set measurement sections through the processing of steps S205 to S207.

In the next step S208, the measurement unit 5234 of the third state characteristic value calculation unit 523 calculates the rutting amount D at the position of the depth map (three-dimensional point group information) corresponding to the measurement position determined by the determination unit 5233. measure. Specifically, as described above, the measurement unit 5234 scans in the road width direction at the determined position on the depth map to obtain depths as shown in FIGS. D ₁ and D ₂ are acquired, and the rutting amount D is measured based on the depth D ₁ and D ₂ and the cross-sectional information obtained by scanning the depth map.

In the next step S<b>209 , the measurement unit 5234 stores the measured rutting amount D in the storage unit 530 .

As described above, the information processing apparatus 50 according to the present embodiment uses a discriminator, which is a learning model obtained by supervised learning (deep learning or machine learning) based on previously added learning data about foreign matter information on a road surface. is used to recognize whether or not a foreign object is included in the subject indicated by the captured image. Then, when determining that there is a foreign object at the determined measurement position, the information processing device 50 sets a position shifted by a predetermined length from the measurement position as a new measurement position, and further determines whether or not there is a foreign object. It is assumed that As a result, the discriminator, which is a learning model, can automatically recognize a foreign object in the captured image, and the process of measuring the amount of rutting or the like in road inspection can be performed accurately and quickly.

Incidentally, in the example of the identification image CI shown in FIG. 19 , road signs, manholes, and the like, which cause an error in the measurement result due to the coating thickness when measuring the rutting amount D, are cited as examples of the foreign matter. , but not limited to these, grass, poles, people, vehicles parked on the road, and the like existing in the measurement section may also be treated as foreign objects and avoided from the measurement position.

Further, the discriminators stored in the storage unit 530 are updated or newly generated by supervised learning (deep learning or machine learning) using new additional information (learning data) about road surface abnormality information. may be Here, although not shown in the functional blocks shown in FIG. 12, the entity that performs learning may be included in the information processing apparatus 50 as a learning unit that performs the above-described learning, or may be performed by an external device. The data of the discriminator (learning model) generated by learning using the learning data may be stored in the storage unit 530 and used.

Further, as described above, in the imaging system 10 according to the first embodiment, the information processing device 50 generates triggers and distributes the generated triggers to all the stereo cameras 6L and 6R. The stereo cameras 6L and 6R are configured to perform imaging in synchronism. At this time, clock generators of the stereo cameras 6L and 6R, trigger distribution components for distributing triggers in the information processing device 50, differences in trigger wiring lengths in the information processing device 50 and the stereo cameras 6L and 6R, etc. Due to the influence, there is a possibility that a slight deviation may occur in the timing at which the stereo cameras 6L and 6R actually capture the trigger and perform imaging. It is preferable to suppress this imaging timing lag as much as possible.

Especially between the stereo cameras 6L and 6R whose baseline lengths are known, it is desirable to minimize the deviation of the imaging timing. For example, when wiring the trigger supply from the camera I/F 5010a to the stereo cameras 6L and 6R, there are cases where a repeater is used to ensure trigger quality, and a photocoupler is used to protect against electrostatic noise. be. In this case, it is desirable to share these repeaters and photocouplers between the stereo cameras 6L and 6R.

[Second embodiment]
Next, the second embodiment will be described, focusing on the differences from the first embodiment. In the above-described first embodiment, supervised learning (deep learning or machine learning) is used for recognition. In this embodiment, the operation of determining whether or not the rutting amount obtained at the measurement position is an abnormal value, and changing the measurement position when it is an abnormal value will be described. The configuration of the imaging system 10 and the hardware configuration of the information processing apparatus according to this embodiment are the same as those of the first embodiment.

(Configuration of functional blocks of information processing device)
FIG. 22 is a diagram showing an example of the configuration of functional blocks for explaining the functions of the information processing device according to the second embodiment. In FIG. It includes an acquisition unit 500 , a UI unit 501 , a control unit 502 and an imaging control unit 503 . The information processing device 50a further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, and a third It includes a state characteristic value calculation unit 523a, a record creation unit 524, a storage unit 530a, and a notification unit 531. Note that the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state The operations of the characteristic value calculation unit 522 and the report creation unit 524 are as described in the first embodiment.

These captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic The value calculation unit 522, the third state characteristic value calculation unit 523a, and the record creation unit 524 are implemented by programs operating on the CPU 5000. FIG. The captured image acquisition unit 500, the UI unit 501, the control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, and the first state characteristic value calculation unit 521 are not limited to these. , the second state characteristic value calculation unit 522, the third state characteristic value calculation unit 523a, and the report creation unit 524 may be partially or entirely configured by hardware circuits that operate in cooperation with each other.

The third state characteristic value calculation unit 523a uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to determine the state for obtaining MCI. Among the characteristic values, a rutting amount D (an example of a physical quantity) is calculated. As shown in FIG. 22, the third state characteristic value calculator 523a has an input section 5231a, a determination section 5233a, a measurement section 5234a, and an output section 5235a.

The input unit 5231a inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 . The stereo captured image input by the input unit 5231a may be any of the stereo captured images captured by the stereo cameras 6L and 6R.

The determination unit 5233a determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point group information (depth map).

The measurement unit 5234a measures the rutting amount D at the measurement position determined by the determination unit 5233a. Then, the measurement unit 5234a determines whether or not the rutting amount D can be measured normally, and whether or not the measured rutting amount D is an abnormal value.

The output unit 5235a outputs the rutting amount D, which is not an abnormal value, measured by the measurement unit 5234a to the report creation unit 524.

The storage unit 530a stores, for example, data used when it is determined whether or not the rutting amount measured by the measuring unit 5234a is an abnormal value. Storage unit 530a is implemented by RAM 5002 or storage 5004 shown in FIG.

Notification unit 531 displays and notifies various information on display 5020 . The notification unit 531 notifies, for example, that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233a, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to the notification by the display on the display 5020, and may be notified by, for example, sound or electronic sound such as an alarm sound.

(Calculation method of rutting amount among road surface property values)
Next, the processing according to the present embodiment will be described in more detail with respect to the processing for measuring the amount of rutting D in step S104 in the flowchart of FIG. 13 described above. FIG. 23 is a flow chart showing an example of a rutting amount measurement process according to the second embodiment.

In step S210, the input unit 5231a of the third state characteristic value calculation unit 523a receives the depth map (three-dimensional point group information) acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. An image (hereinafter simply referred to as a captured image in FIG. 23) is input. Then, the determination unit 5233a of the third state characteristic value calculation unit 523a determines an initial measurement position for measuring the amount of rutting in the captured image input by the input unit 5231a. Then, the determination unit 5233a sets a measurement section surrounded by a predetermined length in each of the vehicle width direction and the traveling direction of the vehicle 1 with each determined initial measurement position as the center in the traveling direction of the vehicle 1 . The specific measurement section setting process is as described in the first embodiment.

In the next step S211, the measurement unit 5234a of the third state characteristic value calculation unit 523a calculates the rutting amount D at the position of the depth map (three-dimensional point group information) corresponding to the measurement position determined by the determination unit 5233a. measure.

In the next step S212, the measurement unit 5234a determines whether or not the rutting amount D has been measured normally. For example, when the rut amount D cannot be measured normally, the portion between the two "ruts" cannot be detected as shown in FIGS _. and _D2 cannot be obtained. If the rutting amount D can be measured normally (step S212: Yes), the process proceeds to step S213, and if it cannot be measured normally (step S212: No), the process proceeds to step S214. When the rutting amount D cannot be measured normally, the notification unit 531 may notify that the rutting amount D cannot be measured normally at the measurement position.

In step S213, when the rutting amount D can be measured normally, the measuring unit 5234a further determines whether or not the rutting amount D is an abnormal value. For example, the measurement unit 5234a compares the measured rutting amount D with the rutting amount D measured in the past or an average rutting amount, or compares the measured rutting amount D within a predetermined normal range. It is sufficient to determine whether or not the value is an abnormal value by determining whether or not the value is at . In this case, data relating to the rutting amount D measured in the past, the average rutting amount, the predetermined normal range, etc. may be stored in advance in the storage unit 530a. If the rutting amount D is not an abnormal value (step S213: Yes), the process proceeds to step S215, and if it is an abnormal value (step S213: No), the process proceeds to step S214.

In step S214, if the rutting amount D measured by the measuring unit 5234a cannot be measured normally or is an abnormal value, the determination unit 5233a moves the measurement position forward or backward in the traveling direction of the vehicle 1 to a predetermined position. It is shifted by a length (for example, by a specific pixel) and determined as a new measurement position (that is, the measurement position is changed). A specific method for determining the new measured value is as described in the first embodiment. After determining the new measurement position, the process returns to step S211.

In step S214, when the new measurement position determined by the determination unit 5233a is out of the measurement section, the measurement unit 5234a may not measure the rutting amount D at the measurement position. In this case, the notification unit 531 may notify that the rutting amount D is not measured at the measurement position.

In step S215, the measuring unit 5234a determines the measured rutting amount D as a normally measured value. Further, the determination unit 5233a determines the measurement position at which the rutting amount D is normally measured by the measurement unit 5234a as the final (regular) measurement position.

In the next step S216, the measurement unit 5234a stores the determined rutting amount D in the storage unit 530a.

As described above, the information processing apparatus 50a according to the present embodiment determines whether or not the rutting amount D measured at the determined measurement position is normally measured, and whether or not it is an abnormal value. . Then, when the measured rutting amount D is not measured normally or is an abnormal value, the information processing device 50a shifts a position by a predetermined length from the current measurement position to a new measurement position. , the rutting amount D is measured again. As a result, it is possible to automatically recognize whether or not the measured rutting amount D is normally measured, so that the measurement processing of the rutting amount and the like in road inspection can be performed accurately and quickly.

[Third Embodiment]
Next, the third embodiment will be described, focusing on the points that are different from the first embodiment. In this embodiment, by comparing abnormality information acquired from an abnormality information database (hereinafter referred to as an abnormality information DB) with the determined measurement position, it is determined whether or not there is a foreign object at the measurement position. In this case, the operation of changing the measurement position will be described. The configuration of the imaging system 10 and the hardware configuration of the information processing apparatus according to this embodiment are the same as those of the first embodiment.

(Configuration of functional blocks of information processing device)
FIG. 24 is a diagram showing an example of the configuration of functional blocks for explaining the functions of the information processing apparatus according to the third embodiment. In FIG. 24 , an information processing device 50 b (an example of a measurement device) includes a captured image acquisition section 500 , a UI section 501 , a control section 502 and an imaging control section 503 . The information processing device 50b further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, and a third It includes a state characteristic value calculation unit 523b, a report creation unit 524, a storage unit 530b, and a notification unit 531. Note that the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state The operations of the characteristic value calculation unit 522 and the report creation unit 524 are as described in the first embodiment.

These captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic The value calculator 522, the third state characteristic value calculator 523b, and the report preparation unit 524 are implemented by programs operating on the CPU 5000. FIG. The captured image acquisition unit 500, the UI unit 501, the control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, and the first state characteristic value calculation unit 521 are not limited to these. , the second state characteristic value calculation unit 522, the third state characteristic value calculation unit 523b, and the report creation unit 524 may be partially or entirely configured by hardware circuits that operate in cooperation with each other.

The third state characteristic value calculation unit 523b uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 to determine the state for obtaining MCI. Among the characteristic values, a rutting amount D (an example of a physical quantity) is calculated. As shown in FIG. 24, the third state characteristic value calculator 523b has an input section 5231b, a determination section 5233b, a measurement section 5234b, and an output section 5235b.

The input unit 5231b inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 . The stereo captured image input by the input unit 5231b may be any of the stereo captured images captured by the stereo cameras 6L and 6R.

The determination unit 5233b determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point group information (depth map).

The measurement unit 5234b measures the rutting amount D at the measurement position determined by the determination unit 5233b.

The output unit 5235b outputs the rutting amount D measured by the measurement unit 5234b to the report creation unit 524. FIG.

The storage unit 530b stores a foreign substance information DB that stores past data and foreign substance information, which is data in which foreign substance positions are specified by visual inspection or the like. Storage unit 530b is implemented by RAM 5002 or storage 5004 shown in FIG.

Notification unit 531 displays and notifies various information on display 5020 . The notification unit 531 notifies, for example, that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233b, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to the notification by the display on the display 5020, and may be notified by, for example, sound or electronic sound such as an alarm sound.

(Calculation method of rutting amount among road surface property values)
Next, the processing according to the present embodiment will be described in more detail with respect to the processing for measuring the amount of rutting D in step S104 in the flowchart of FIG. 13 described above. FIG. 25 is a flow chart showing an example of a rutting amount measurement process according to the third embodiment.

In step S220, the input unit 5231b of the third state characteristic value calculation unit 523b receives the depth map (three-dimensional point group information) acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. An image (hereinafter simply referred to as a captured image in FIG. 25) is input. Then, the determination unit 5233b of the third state characteristic value calculation unit 523b determines an initial measurement position for measuring the amount of rutting in the captured image input by the input unit 5231b. Then, the determination unit 5233b sets a measurement section surrounded by a predetermined length in each of the vehicle width direction and the traveling direction of the vehicle 1 with each determined initial measurement position as the center in the traveling direction of the vehicle 1 . The specific measurement section setting process is as described in the first embodiment.

In the next step S221, the input unit 5231b further extracts and inputs abnormality information corresponding to the input captured image from the abnormality information DB stored in the storage unit 530b.

In the next step S222, the determination unit 5233b determines whether or not the foreign object is avoided at the determined measurement position based on the result of comparison between the captured image input by the input unit 5231b and the abnormality information extracted from the abnormality information DB. do. If the foreign matter is avoided (step S222: Yes), the process proceeds to step S224, and if the foreign matter is not avoided (step S222: No), the process proceeds to step S223. If the foreign object is not avoided, the notification unit 531 may notify that the rutting amount D cannot be measured at the measurement position.

In step S223, if a foreign object exists at the determined measurement position in the captured image, the determination unit 5233b moves the measurement position forward or backward in the traveling direction of the vehicle 1 by a predetermined length (for example, a specific pixel length). , and determined as a new measurement position (that is, change the measurement position). A specific method for determining the new measured value is as described in the first embodiment. After determining the new measurement position, the process returns to step S222.

In step S223, when the new measurement position determined by the determination unit 5233b is out of the measurement section, the measurement unit 5234b may not measure the rutting amount D at the measurement position. In this case, the notification unit 531 may notify that the rutting amount D is not measured at the measurement position.

In step S224, the determination unit 5233b determines the measurement position as a normal measurement position for measuring the rutting amount D if no foreign matter exists at the measurement position determined in the captured image. The determination unit 5233b thus determines the measurement position for measuring the rutting amount D in each of the set measurement sections through the processing of steps S222 to S224.

In the next step S225, the measurement unit 5234b of the third state characteristic value calculation unit 523b calculates the rutting amount D at the position of the depth map (three-dimensional point group information) corresponding to the measurement position determined by the determination unit 5233b. measure. Specifically, as described above, the measurement unit 5234b scans in the road width direction at the determined position on the depth map to obtain depths as shown in FIGS. D ₁ and D ₂ are acquired, and the rutting amount D is measured based on the depth D ₁ and D ₂ and the cross-sectional information obtained by scanning the depth map.

In the next step S226, the measurement unit 5234b saves the measured rutting amount D in the storage unit 530b.

As described above, the information processing apparatus 50b according to the present embodiment extracts abnormality information corresponding to a captured image from a foreign substance information DB that stores in advance foreign substance information, which is past data or data in which the position of a foreign substance is identified by visual observation or the like. Based on the abnormality information, it is determined whether or not the foreign object is avoided at the determined measurement position. When the information processing device 50b determines that there is a foreign object at the determined measurement position, the information processing device 50b sets a position shifted by a predetermined length from the measurement position as a new measurement position, and further determines whether or not there is a foreign object. It is assumed that As a result, it is possible to automatically recognize a foreign object in a captured image based on the abnormality information accumulated in the abnormality information DB, and to accurately and quickly perform measurement processing such as the amount of rutting in a road inspection.

REFERENCE SIGNS LIST 1 vehicle 2 mounting portion 3 mounting member 4 road surface 5 PC
6, 6C, _6L , 6R Stereo camera 6CL, 6C _R imaging lens 6L _L , 6LR imaging lens _6RL , _{6R R imaging} lens 10 Imaging system 50, 50a, 50b Information processing device 60C, 60L, 60R Stereo imaging range 60L ', 60R _' Stereo imaging range _60CL , 60C _R imaging range _60LL , 60LR _R imaging range _60LL ', 60LR _' imaging range 60RL, 60RR _R imaging range 61 area ₁₀₀₁ , ₁₀₀₂ imaging unit ₁₀₁₁ , 101 ₂ imaging control unit 102 velocity acquisition unit 103 generation unit 400a, 400b camera 401 lens 402 imaging element 403 target object 500 captured image acquisition unit 501 UI unit 502 control unit 503 imaging control unit 510 matching processing unit 511 3D information generation unit 520 3D information Acquisition unit 521 First state characteristic value calculation unit 522 Second state characteristic value calculation unit 523, 523a, 523b Third state characteristic value calculation unit 524 Record creation unit 530, 530a, 530b Storage unit 531 Notification unit _600L , _600R Imaging Optical system _601L , _601R Imaging device _602L , _602R Driving unit _603L , _603R Signal processing unit 604 Output unit 901 Manhole unit 902 White line unit 903 Road sign unit 904 Road shoulder 905 Others 5000 CPU
5001 ROMs
5002 RAMs
5003 Graphics I/F
5004 storage 5005 input device 5006 data I/F
5007 Communication I/F
5010 Camera I/F
5020 display 5021 speed acquisition unit 5231, 5231a, 5231b input unit 5232 recognition unit 5233, 5233a, 5233b determination unit 5234, 5234a, 5234b measurement unit 5235, 5235a, 5235b output unit CI identification image

JP 2015-197804 A

In order to solve the above-described problems and achieve the object, the present invention provides a determination unit that determines a measurement position for measuring a predetermined physical quantity with respect to an object to be measured based on an image captured by an imaging unit; a measurement unit that measures the physical quantity for the object to be measured at the measurement position determined by the determination unit, and the determination unit measures the physical quantity measured by the measurement unit if the physical quantity is not abnormal. and determining the measurement position as a regular measurement position .

Claims (6)

a recognition unit that recognizes the state of the object to be measured in the captured image using a classifier obtained by learning using learning data related to the state of the object to be measured;
a determination unit that determines a measurement position for measuring a predetermined physical quantity with respect to the object to be measured based on the recognition result of the state of the object to be measured by the recognition unit;
a measurement unit that measures the physical quantity with respect to the object to be measured at the measurement position determined by the determination unit;
with
The measurement device, wherein the measurement unit does not measure the physical quantity of the object to be measured when the measurement position determined by the determination unit is out of a predetermined range. The determination unit avoids the foreign object when determining that a foreign object different from the object to be measured exists at the determined measurement position based on the recognition result of the state of the object to be measured by the recognition unit. 2. The measuring device according to claim 1, wherein the position is determined to a new measuring position. The measuring device according to claim 1 or 2, wherein the recognition unit recognizes the state of the object in the captured image using the classifier obtained by the learning based on new additional information. 4. The method according to any one of claims 1 to 3, further comprising a notification unit that notifies that the measurement unit does not measure the physical quantity of the object to be measured when the measurement position determined by the determination unit is out of a predetermined range. A measuring device according to any one of the preceding paragraphs. an imaging unit that captures an image of an object to be measured to obtain a captured image;
a recognition unit that recognizes the state of the object to be measured in the captured image using a classifier obtained by learning using learning data on the state of the object to be measured;
a determination unit that determines a measurement position for measuring a predetermined physical quantity with respect to the object to be measured based on the recognition result of the state of the object to be measured by the recognition unit;
a measurement unit that measures the physical quantity with respect to the object to be measured at the measurement position determined by the determination unit;
has
The measurement system, wherein the measurement unit does not measure the physical quantity of the object to be measured when the measurement position determined by the determination unit is out of a predetermined range. A measuring device according to any one of claims 1 to 4;
an imaging unit that captures the captured image;
with
The vehicle, wherein the imaging unit is a plurality of stereo cameras.

Images