JP7480833B2 - Measuring equipment, measuring systems and vehicles - Google Patents

Description

The present invention relates to a measuring device, a measuring system, and a vehicle.

Since July 2014, close-up visual inspections have been tightened in the maintenance and inspection of infrastructure such as highways, bridges, and tunnels, and full close-up visual inspections are now required once every five years. For example, when inspecting the condition of paved roads (hereafter simply referred to as "roads"), it is already known that inspections are conducted using cracks, rutting (unevenness in the width direction of the road), and flatness (unevenness in the direction of vehicle travel) as indicators of typical damage.

However, current road inspections are carried out manually, which means there is a risk of human error during the inspection. In addition, when manual inspection is required, depending on the inspection item, there are some areas that must be avoided, such as signs on the road surface, which is very time-consuming.

One such technology for measuring rutting during road inspections is disclosed in which a special vehicle is used, a line scan camera, laser light, GPS (Global Positioning System) and audio are recorded, operation is controlled by a PC (Personal Computer), rutting is measured with a measuring unit (laser light), and an inspection screen showing the measurement position of the measurement information is displayed (for example, Patent Document 1).

However, the technology described in Patent Document 1 has the problem that it is not possible to avoid areas where foreign objects such as road signs must be avoided when measuring ruts.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a measuring device, measuring system, and vehicle that can perform measurement processing in inspection of an object to be measured accurately and quickly by distinguishing between foreign objects that are different from the object to be measured.

In order to solve the above-mentioned problems and achieve the object, the present invention comprises a determination unit that determines a measurement position at which a predetermined physical quantity of a measured object is measured based on an image captured by an imaging unit, and a measurement unit that measures the physical quantity of the measured object at the measurement position determined by the determination unit, wherein if the physical quantity measured by the measurement unit is not abnormal, the determination unit determines the measurement position at which the physical quantity is measured as a normal measurement position, and if the measurement position determined by the determination unit is outside a predetermined range, the measurement unit does not measure the physical quantity of the measured object .

According to the present invention, by distinguishing foreign objects that are different from the object to be measured, the measurement process during inspection of the object to be measured can be performed accurately and quickly.

FIG. 1 is a diagram for explaining measurement of the amount of rutting. FIG. 2 is a diagram for explaining the formation of flatness. FIG. 3 is a diagram illustrating an example of the configuration of an imaging system according to the first embodiment. FIG. 4 is a diagram for explaining an imaging range in the traveling direction of the vehicle captured by the stereo camera according to the first embodiment. FIG. 5 is a diagram for explaining an imaging range of a vehicle in a road width direction by the stereo camera according to the first embodiment. FIG. 6 is a diagram showing the overlap of stereo image capture ranges captured by the stereo cameras according to the first embodiment in the traveling direction of the vehicle. FIG. 7 is a diagram illustrating an example of an imaging system including three stereo cameras according to the first embodiment. FIG. 8 is a block diagram showing a schematic overall configuration of the imaging system according to the first embodiment. FIG. 9 is a diagram illustrating an example of a schematic hardware configuration of the imaging system according to the first embodiment. FIG. 10 is a diagram illustrating an example of a hardware configuration of the stereo camera according to the first embodiment. FIG. 11 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the first embodiment. FIG. 12 is a diagram illustrating an example of a functional block configuration for explaining functions of the information processing device according to the first embodiment. FIG. 13 is a flowchart illustrating an example of the flatness calculation process according to the first embodiment. FIG. 14 is a flowchart illustrating an example of a depth map generation process according to the first embodiment. FIG. 15 is a diagram for explaining trigonometry. FIG. 16 is a flowchart illustrating an example of the process of estimating the camera position and orientation according to the first embodiment. FIG. 17 is a diagram for explaining the process of estimating the camera position and orientation according to the first embodiment. FIG. 18 is a flowchart illustrating an example of a process for measuring the amount of rutting according to the first embodiment. FIG. 19 is a diagram illustrating an example of an image showing a result of foreign matter recognition according to the first embodiment. FIG. 20 is a diagram for explaining the measurement positions of the rutting amount. FIG. 21 is a diagram showing measurement positions of the amount of rutting. FIG. 22 is a diagram illustrating an example of a functional block configuration for explaining functions of an information processing device according to the second embodiment. FIG. 23 is a flowchart illustrating an example of a process for measuring the amount of rutting according to the second embodiment. FIG. 24 is a diagram illustrating an example of a functional block configuration for explaining functions of an information processing device according to the third embodiment. FIG. 25 is a flowchart illustrating an example of a process for measuring the amount of rutting according to the third embodiment.

Below, with reference to the drawings, embodiments of the measuring device, measuring system, and vehicle according to the present invention will be described in detail. Furthermore, the present invention is not limited to the following embodiments, and the components in the following embodiments include those that a person skilled in the art would easily come up with, those that are substantially the same, and those that are within the scope of what is called equivalent. Furthermore, various omissions, substitutions, modifications, and combinations of the components can be made without departing from the spirit of the following embodiments.

[Outline of existing road condition inspection methods]
Prior to describing the embodiments, an overview of an existing road condition inspection method will be given. As an index for evaluating road conditions, a maintenance control index (MCI) has been established. The MCI quantitatively evaluates the serviceability of a pavement based on three road surface property values: "crack rate,""amount of rutting," and "flatness."

Of these, the "crack rate" is calculated by, for example, dividing the road surface into 50 cm meshes, calculating the crack area according to the number of cracks and patching area as shown below, and applying the calculation result to the following formula (1) to find the crack rate.

1 crack ⇒ 0.15 [m ² ] crack 2 cracks ⇒ 0.25 [m ² ] crack Patching area 0-25 [%] ⇒ 0 cracks [m ² ]
Patching area 25-75% ⇒ Crack ^0.125m2
Patching area 75% or more ⇒ Crack 0.25 ^m2

As shown in Fig. 1(a) and Fig. 1(b), for example, the depths _D1 and _D2 of two "ruts" that occur per lane are measured, and the larger of the measured depths _D1 and _D2 is used as the "amount of rutting." There are several patterns of how "ruts" are dug, so a measurement method suited to each pattern is adopted. Fig. 1(a) shows an example of a measurement method when the part between two "ruts" is higher than both ends of the two "ruts," and Fig. 1(b) shows an example of a measurement method when the part between two "ruts" is lower than both ends of the two "ruts."

For "flatness", the height from the road surface is measured at three points at intervals of 1.5 m along the direction of travel of the vehicle. For example, as shown in FIG. 2, measuring instruments are provided at three points A, B, and C at intervals of 1.5 m on the underside of the vehicle to measure heights _X1 , _X2 , and _X3 . Based on the measured heights _X1 , _X2 , and _X3 , the displacement amount d is calculated using the following formula (2). This measurement is performed multiple times while moving the vehicle. Based on the multiple displacement amounts d obtained in this way, the flatness σ is calculated using the following formula (3).

The above-mentioned "crack rate," "amount of rutting," and "flatness" measurements are performed for each evaluation section of, for example, 100 m. If the damaged areas are known in advance, measurements may be performed by dividing the 100 m into smaller units, such as 40 m + 60 m. The MCI is calculated based on the results of measurements performed in 100 m increments, and a report is prepared.

The MCI is calculated using the four formulas shown in Table 1 below based on the measured crack rate C [%], rutting depth D [mm], and flatness σ [mm] to calculate the values MCI, MCI1, MCI2, and MCI3. The minimum of the calculated values MCI, MCI1, MCI2, and MCI3 is then adopted as the MCI. Based on the adopted MCI, an evaluation is performed according to the evaluation criteria shown in Table 2 below to determine whether or not the road surface in the evaluation section requires repair.

[First embodiment]
Next, an imaging system according to a first embodiment will be described. In the first embodiment, a stereo camera capable of simultaneously capturing images of an object from a plurality of different directions to obtain two captured images (hereinafter, sometimes referred to as "stereo captured images") and obtaining depth information is attached to a vehicle to capture an image of a road surface. Depth information relative to the road surface is obtained from the imaging position based on the captured stereo captured images, a three-dimensional shape of the road surface is generated, and three-dimensional road surface data is created. By analyzing this three-dimensional road surface data, the "crack rate,""ruttingamount," and "flatness" used to calculate the MCI can be obtained.

A more specific explanation will be given. A stereo camera has two cameras spaced a predetermined distance apart (called the baseline length), and outputs a pair of captured images (stereo captured images) captured by the two cameras. By searching for corresponding points between the two captured images contained in the stereo captured images, it is possible to restore the depth distance for any point in the captured images. Data that restores the depth distance (hereinafter sometimes referred to as "depth value") for the entire captured image and represents each pixel with depth information is called a depth map. In other words, a depth map is three-dimensional point cloud information consisting of a collection of points, each of which has three-dimensional information.

This stereo camera is attached facing downwards in one location, such as the rear of the vehicle, so that it can capture images of the ground, and the vehicle is moved along the road to be measured. For the sake of simplicity, the stereo camera mounted on the vehicle for measurement has an imaging range that covers a specified length in the road width direction.

The rutting amount D is calculated by arranging the depth values of the striped cut-out portions in the road width direction in the depth map restored from the stereo images captured by the stereo cameras. The depths _D1 and _D2 can be calculated based on the change in the depth values in the road width direction.

The crack rate C is obtained by analyzing the captured image of the road surface to detect "cracks" and the like, and then performing calculations using the above-mentioned formula (1) based on the detection results.

Here, since the imaging range is limited in the vehicle's travel direction in a single imaging, it is not possible to calculate the crack rate C of a 100 m section, for example, based on the image captured in a single imaging. Therefore, while moving the vehicle along the road, images are captured sequentially for each movement according to the length of the imaging range in the vehicle's travel direction. At this time, the imaging trigger (imaging trigger) is controlled so that the imaging range in the previous imaging and the imaging range in the current imaging overlap at a pre-designed overlap rate or higher.

In this way, by controlling the image capture trigger according to the movement of the vehicle, it is possible to capture images of the road surface to be measured without missing anything. Therefore, images captured sequentially as the vehicle moves are stitched together using an image process called stitching to generate a single image that includes an image of the road surface over a 100 m section, for example. The crack rate C on the road surface can be measured by visually checking or analyzing this image.

Flatness is measured using a technique called Structure from Motion (SfM), which estimates the imaging position based on images taken from different locations with sufficient overlap.

Below is an overview of the SfM process. First, images captured with overlapping imaging ranges are used, and points in each image that capture the same point are detected as corresponding points. It is desirable to detect as many corresponding points as possible. Next, for example, the movement of the camera from the imaging point of the first image to the imaging point of the second image is calculated by formulating simultaneous equations using the coordinates of the detected corresponding points with rotation and translation as unknown parameters, and parameters that minimize the total error are found. In this way, the imaging position of the second image can be calculated.

As described above, the depth of any point in the image can be restored as a depth map from the stereo images captured at each imaging point. For the depth map corresponding to the first stereo image, the depth map corresponding to the second stereo image is coordinate-converted into a depth map whose origin is the imaging position of the first camera calculated using the simultaneous equations described above. This allows the two depth maps to be unified into the coordinate system of the first depth map. In other words, the two depth maps can be synthesized to generate one depth map.

By performing this process on depth maps based on all stereo images captured in a 100 m section, for example, and synthesizing them, the road surface of the 100 m section is restored in a single three-dimensional space. The depth value of the road surface restored in this way can be applied to the above-mentioned equation (2) to calculate the displacement amount d. The flatness σ is calculated by applying this displacement amount d to the above-mentioned equation (3).

In the first embodiment, the flatness σ of the road surface, the amount of rutting D, and the crack rate C for calculating the MCI can be measured all at once by capturing images using a stereo camera mounted on a vehicle and performing image processing on the captured stereo image. In the first embodiment, this measurement can be performed using an imaging system including a stereo camera and an information processing device that performs image processing on the stereo image, making it possible to calculate the MCI with a simple configuration.

In addition, in existing technologies, measurements related to the three indices for calculating the MCI are performed using separate devices, which makes data storage and control and management of time synchronization between the data cumbersome. In contrast, in the first embodiment, the three indices for calculating the MCI are calculated based on a common stereo image, which makes it easier to store data and control and management of time synchronization between the data.

(Stereo camera placement)
Next, an example of a stereo camera arrangement in the first embodiment will be described. Fig. 3 is a diagram showing an example of the configuration of the imaging system according to the first embodiment. Fig. 3(a) is a diagram showing a state in which the imaging system according to the first embodiment is mounted on the vehicle 1, as viewed from the side of the vehicle 1. In Fig. 3(a), the direction toward the left end of the figure is the traveling direction of the vehicle 1. That is, in Fig. 3(a), the left end side of the vehicle 1 is the front part of the vehicle 1, and the right end side is the rear part of the vehicle 1. Fig. 3(b) is a diagram showing an example of the vehicle 1 as viewed from the rear side.

The imaging system according to the first embodiment has a mounting member 3 with a mounting section 2 for mounting an imaging device fixed to the rear of a vehicle 1, which is a moving body on which the imaging system is mounted, and one or more stereo cameras 6 are attached to the mounting section 2. Here, as illustrated in FIG. 3(b), two stereo cameras 6L and 6R (imaging sections) are attached to both ends of the vehicle 1 in the width direction of the body. Each of the stereo cameras 6L and 6R is attached in a direction to capture an image of a road surface 4 (an example of an object to be measured) on which the vehicle 1 is moving. Preferably, each of the stereo cameras 6L and 6R is attached so as to capture an image of the road surface 4 from a vertical direction.

Hereinafter, when there is no need to distinguish between the stereo cameras 6L and 6R, the stereo cameras 6L and 6R will be collectively described as the stereo camera 6.

The stereo camera 6 is controlled by, for example, a personal computer (PC) 5 installed inside the vehicle 1. The worker operates the PC 5 to instruct the stereo camera 6 to start capturing images. When the instruction to start capturing images is given, the PC 5 starts capturing images with the stereo camera 6. The capturing images is timed and repeatedly performed according to the moving speed of the stereo camera 6, i.e., the vehicle 1. Note that the device that controls the stereo camera 6 is not limited to the PC 5, and may be a workstation or a dedicated device specialized for controlling the stereo camera 6.

For example, when imaging of a required section is completed, the operator operates the PC 5 to instruct the end of imaging. In response to the instruction to end imaging, the PC 5 ends imaging by the stereo camera 6.

Figures 4 and 5 are diagrams showing an example of the imaging range of the stereo camera 6 that can be applied to the first embodiment. Note that in Figures 4 and 5, parts that are common to the above-mentioned Figure 3 are given the same reference numerals, and detailed explanations are omitted.

Figure 4 is a diagram for explaining the imaging range in the traveling direction of the vehicle by the stereo camera according to the first embodiment (referred to as the traveling direction field of view Vp). Note that in Figure 4, as in Figure 1 described above, the direction toward the left end of the figure is the traveling direction of the vehicle 1. As shown in Figure 4, the traveling direction field of view Vp is determined according to the angle of view α of the stereo camera 6 and the height h of the stereo camera 6 relative to the road surface 4.

Figure 5 is a diagram for explaining the imaging range of the vehicle in the road width direction by the stereo camera according to the first embodiment. Figure 5 is a view of the vehicle 1 seen from the rear side, and parts common to Figure 3 (b) are given the same reference numerals and detailed explanations are omitted.

5(a), the stereo camera 6L includes two imaging lenses _6LL and _6LR . A line connecting the imaging lenses _6LL and _6LR is called a baseline, and its length is called a baseline length. The stereo camera 6L is disposed so that the baseline is perpendicular to the traveling direction of the vehicle 1. Similarly, the stereo camera 6R includes two imaging lenses _6RL and _6RR separated by the baseline length, and is disposed so that the baseline is perpendicular to the traveling direction of the vehicle 1.

5B shows an example of the imaging ranges of the stereo cameras 6L and 6R of the first embodiment. In the stereo camera 6L, the imaging ranges 60L _L and 60L _R of the imaging lenses 6L _L and 6L _R are overlapped with a shift according to the baseline length and height h. Similarly, in the stereo camera 6R, the imaging ranges 60R _L and 60R _R of the imaging lenses 6R _L and 6R _R are overlapped with a shift according to the baseline length and height h.

Hereinafter, unless otherwise specified, these imaging ranges 60L _L and 60L _R and imaging ranges 60R _L and 60R _R are collectively referred to as stereo imaging ranges 60L and 60R, respectively. As shown in Fig. 5(b), the stereo cameras 6L and 6R are arranged such that a partial area of the stereo imaging range 60L at one end side in the width direction of the vehicle 1 and a partial area of the stereo imaging range 60R at the other end side in the width direction overlap with each other at a predetermined overlap rate in an area 61.

6 is a diagram showing the overlap of the stereo imaging range 60L by the stereo camera 6L in the traveling direction of the vehicle 1, which is applicable to the first embodiment. In FIG. 6, the left camera field of view V _L and the right camera field of view V _R respectively indicate the fields of view (imaging ranges) of the imaging lenses 6L _L and 6L _R in the stereo camera 6L. It is assumed that the stereo camera 6L captures images twice in a moving vehicle 1. The stereo camera 6L captures images of the stereo imaging range 60L (imaging ranges 60L _L and 60L _R ) the first time, and captures images of the stereo imaging range 60L' (imaging ranges 60L _L ' and 60L _R ') that have moved in the traveling direction of the vehicle 1 by a distance corresponding to the moving distance of the vehicle 1 relative to the stereo imaging range 60L.

At this time, the imaging timing of the stereo camera 6L is controlled so that the stereo imaging range 60L and the stereo imaging range 60L' overlap with respect to the traveling direction field of view Vp at an overlap rate equal to or greater than a predetermined traveling direction overlap rate Dr. In this way, by sequentially capturing images of the stereo imaging ranges 60L and 60L' along the time axis, it is possible to facilitate the process of stitching together two stereo images captured of the stereo imaging ranges 60L and 60L'.

In the above description, the imaging system according to the first embodiment has been described as using two stereo cameras 6L and 6R, but this is not limited to this example. For example, the imaging system according to the first embodiment may be configured using three stereo cameras 6L, 6R, and 6C by adding one more stereo camera 6C to the stereo cameras 6L and 6R, as shown in FIG. 7(a).

In the example of Fig. 7(a), the interval between the stereo cameras 6L and 6R is wider than when only the stereo cameras 6L and 6R are used, and the stereo camera 6C is disposed in the center. As shown in Fig. 7(b), the imaging ranges 60C _L and 60C _R of the imaging lenses 6C _L and 6C _R of the stereo camera 6C form a stereo imaging range 60C. The stereo cameras 6L, 6C, and 6R are disposed so that the stereo imaging ranges 60L, 60C, and 60R of each of them overlap with each other in the width direction of the vehicle 1 at a predetermined overlap rate.

In this way, by using three stereo cameras 6L, 6C, and 6R to capture an image of one lane, it is possible to set the image capture range to the right, center, and left side of the lane, respectively, and capture high-quality (high-resolution) stereo images with a small number of stereo cameras. Here, the road width in particular is generally specified as 3.5 m. Therefore, to accommodate this road width of 3.5 m, it is conceivable to use stereo cameras 6L and 6R to capture both ends of the lane in the road width direction, and stereo camera 6C to capture the center.

The imaging system may be configured using only one stereo camera whose imaging range has an angle of view that can include the specified road width (3.5 m).

(Configuration of imaging system)
Next, a configuration of an image capturing system according to the first embodiment will be described below, assuming that the image capturing system includes two stereo cameras.

8 is a block diagram showing an example of a schematic configuration of an imaging system according to the first embodiment. In FIG. 8, the imaging system 10 includes imaging units _100-1 and _100-2 , imaging control units _101-1 and _101-2 , a velocity acquisition unit 102, and a generation unit 103.

The imaging units _100.1 and _100.2 correspond to the stereo cameras 6L and 6R described above, respectively. The imaging control units _101.1 and _101.2 control the imaging operations of the imaging units _100.1 and _100.2 , such as the imaging timing, exposure, and shutter speed. The speed acquisition unit 102 acquires the speed of the imaging units _100.1 and _100.2 relative to the subject (road surface 4). The generation unit 103 generates a trigger (imaging trigger) for instructing the imaging units _100.1 and _100.2 to capture images based on the speed acquired by the speed acquisition unit 102 and the traveling direction field of view Vp. The generation unit 103 sends the generated trigger to the imaging control units _101.1 and _101.2 . The imaging control units _101.1 and _101.2 cause the imaging units _100.1 and _100.2 to perform imaging operations in response to the trigger sent from the generation unit 103.

9 is a diagram showing an example of a schematic hardware configuration of an imaging system according to the first embodiment. In FIG. 9, the imaging system 10 includes stereo cameras 6L and 6R, and an information processing device 50 (an example of a measurement device) corresponding to the PC 5 in FIG. 3. The information processing device 50 generates a trigger at a predetermined timing and sends the generated trigger to the stereo cameras 6L and 6R. The stereo cameras 6L and 6R capture images in response to the trigger. Each stereo captured image captured by the stereo cameras 6L and 6R is transmitted to the information processing device 50. The information processing device 50 stores and accumulates each stereo captured image received from the stereo cameras 6L and 6R in a storage device or the like. The information processing device 50 performs image processing such as generating a depth map and stitching together the generated depth maps based on the accumulated stereo captured images.

Figure 10 is a diagram showing an example of the hardware configuration of the stereo camera 6L according to the first embodiment. Note that the stereo camera 6R can be realized with a similar configuration to the stereo camera 6L.

10, the stereo camera 6L includes imaging optical systems _600L and _600R , imaging elements _601L and _601R , driving units _602L and _602R , signal processing units _603L and _603R , and an output unit 604. Of these, the imaging optical system _600L corresponds to the imaging lens 6L _L described above. Similarly, the imaging optical system _600R corresponds to the imaging lens 6L _R described above.

The imaging optical system _600L is an optical system having an angle of view α and a focal length f, and projects light from a subject onto the imaging element _601L . The imaging element _601L is, for example, an optical sensor using a complementary metal oxide semiconductor (CMOS), and outputs a signal according to the projected light. An optical sensor using a charge coupled device (CCD) may be applied to the imaging element _601L . The driving unit _602L drives the imaging element _601L , and outputs the signal output from the imaging element _601L after performing predetermined processing such as noise removal and gain adjustment. The signal processing unit _603L performs A/D conversion on the signal output from the driving unit _602L to convert the signal into a digital image signal (captured image). The signal processing unit _603L performs predetermined image processing such as gamma correction on the converted image signal and outputs it. The captured image output from the signal processing unit 603 L is sent to the output unit 604 .

The operations of the imaging optical system 600 _R , imaging element 601 _R , drive section 602 _R , and signal processing section 603 _R are similar to those of the imaging optical system 600 _L , imaging element 601 _L , drive section 602 _L , and signal processing section 603 _L described above.

For example, a trigger output from the information processing device 50 is transmitted to the driving units _602L and _602R . The driving units _602L and _602R take in signals from the image pickup elements _601L and _601R at the timing of the received trigger, and perform imaging.

Here, the driving units _602L and _602R perform exposure in the image pickup elements _601L and _601R by a batch simultaneous exposure method. This method is called a global shutter. In contrast, the rolling shutter is a method of capturing light in order (line order) from the top of the pixel position, so each line in a frame does not capture the subject at the exact same time. In the case of the rolling shutter method, if the camera or the subject moves at high speed while capturing the image signal of one frame, the image of the subject will be captured with a shift according to the line position. Therefore, in the stereo cameras 6L and 6R according to the first embodiment, a global shutter is used to capture the road shape correctly in terms of projection geometry.

The output unit 604 outputs, as a pair of stereo captured images, the captured images of each frame supplied from the signal processing units 603 _L and 603 _R. The stereo captured images output from the output unit 604 are sent to the information processing device 50 and stored therein.

Fig. 11 is a diagram showing an example of the hardware configuration of an information processing device according to the first embodiment. In Fig. 11, the information processing device 50 includes a CPU (Central Processing Unit) 5000, a ROM (Read Only Memory) 5001, a RAM (Random Access Memory) 5002, a graphics I/F (interface) 5003, a storage 5004, an input device 5005, a data I/F 5006, and a communication I/F 5007, each of which is connected to a bus 5030. Furthermore, the information processing device 50 includes a camera I/F 5010 and a speed acquisition unit 5021, each of which is connected to the bus 5030.

Storage 5004 is a storage medium that stores data in a non-volatile manner, and may be a hard disk drive or flash memory. Storage 5004 stores programs and data for the operation of CPU 5000.

The CPU 5000 uses the RAM 5002 as a work memory and controls the overall operation of the information processing device 50 according to a program previously stored in at least one of the ROM 5001 and the storage 5004, for example. The graphics I/F 5003 generates a display signal that the display 5020 can handle, based on a display control signal generated by the CPU 5000 according to the program. The display 5020 displays a screen according to the display signal supplied from the graphics I/F 5003.

The input device 5005 receives a user operation and outputs a control signal corresponding to the received user operation. The input device 5005 may be a pointing device such as a mouse or a tablet, or a keyboard. The input device 5005 and the display 5020 may be integrally formed to form a so-called touch panel configuration.

The data I/F 5006 transmits and receives data to and from external devices. For example, a USB (Universal Serial Bus) can be used as the data I/F 5006. The communication I/F 5007 controls communication with an external network according to instructions from the CPU 5000.

The camera I/F 5010 is an interface for each of the stereo cameras 6L and 6R. Each stereo captured image output from each of the stereo cameras 6L and 6R is passed to, for example, the CPU 5000 via the camera I/F 5010. The camera I/F 5010 also generates the above-mentioned trigger according to an instruction from the CPU 5000, and sends the generated trigger to each of the stereo cameras 6L and 6R.

The speed acquisition unit 5021 acquires speed information indicating the speed of the vehicle 1. When the stereo cameras 6L and 6R are attached to the vehicle 1, the speed information acquired by the speed acquisition unit 5021 indicates the speed of each of the stereo cameras 6L and 6R relative to the subject (road surface). The speed acquisition unit 5021 has a function of receiving, for example, a GNSS (Global Navigation Satellite System) signal, and acquires speed information indicating the speed of the vehicle 1 based on the Doppler effect of the received GNSS signal. Not limited to this, the speed acquisition unit 5021 can also acquire the speed information directly from the vehicle 1.

Fig. 12 is a diagram showing an example of a functional block configuration for explaining the functions of the information processing device according to the first embodiment. In Fig. 12, the information processing device 50 includes an image acquisition unit 500, a UI unit 501, a control unit 502, and an image capture control unit 503. The information processing device 50 further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, a third state characteristic value calculation unit 523, a report creation unit 524, a storage unit 530, and an alarm unit 531.

The captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523, and report creation unit 524 are realized by a program that runs on the CPU 5000. Without being limited to this, some or all of the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523, and report creation unit 524 may be configured by hardware circuits that operate in cooperation with each other.

The captured image acquisition unit 500 acquires stereo captured images from each of the stereo cameras 6L and 6R. The captured image acquisition unit 500 stores the acquired stereo captured images, for example, in the storage 5004. The captured image acquisition unit 500 also acquires the stored stereo captured images, for example, from the storage 5004.

The UI unit 501 realizes a user interface by displaying on the input device 5005 and the display 5020. The control unit 502 controls the operation of the entire information processing device 50.

The imaging control unit 503 corresponds to the above-mentioned imaging control units _101.1 and _101.2 , the speed acquisition unit 102, and the generation unit 103. That is, the imaging control unit 503 acquires speed information indicating the speed of each of the stereo cameras 6L and 6R relative to the subject (road surface 4), and generates a trigger for instructing each of the stereo cameras 6L and 6R to capture an image based on the acquired speed information and the angle of view α and height h of each of the stereo cameras 6L and 6R that are set in advance.

The matching processing unit 510 performs matching processing using two captured images constituting a stereo captured image acquired by the captured image acquisition unit 500. The 3D information generation unit 511 performs processing related to three-dimensional information. For example, the 3D information generation unit 511 uses the results of the matching processing by the matching processing unit 510 to obtain depth information by trigonometry or the like, and generates three-dimensional point cloud information (depth map) based on the obtained depth information.

The 3D information acquisition unit 520 acquires the three-dimensional point cloud information obtained for each stereo image by the 3D information generation unit 511.

The first state characteristic value calculation unit 521 uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500 to calculate the crack rate C, which is one of the state characteristic values for determining the MCI.

The second state characteristic value calculation unit 522 calculates the flatness σ, one of the state characteristic values for determining the MCI, using the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500.

The third state characteristic value calculation unit 523 calculates the rutting amount D (an example of a physical quantity) from among the state characteristic values for determining the MCI, using the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. As shown in FIG. 12 , the third state characteristic value calculation unit 523 has an input unit 5231, a recognition unit 5232, a determination unit 5233, a measurement unit 5234, and an output unit 5235.

The input unit 5231 inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. The stereo image input by the input unit 5231 may be any one of the stereo images captured by the stereo cameras 6L and 6R.

The recognition unit 5232 uses a classifier, which is a learning model obtained by supervised learning (deep learning or machine learning) based on learning data about foreign object information on the road surface that has been added in advance, to recognize whether or not a foreign object is included in the subject shown in the stereo image input by the input unit 5231. At this time, the recognition unit 5232 reads out the classifier data from the storage unit 530.

The determination unit 5233 determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point cloud information (depth map) based on the result of the recognition of foreign objects in the stereoscopic captured images by the recognition unit 5232.

The measurement unit 5234 measures the amount of rutting D at the measurement position in the three-dimensional point cloud information determined by the determination unit 5233.

The output unit 5235 outputs the rutting amount D measured by the measurement unit 5234 to the report creation unit 524.

The report creation unit 524 calculates the MCI based on the state characteristic values calculated by the first state characteristic value calculation unit 521, the second state characteristic value calculation unit 522, and the third state characteristic value calculation unit 523, and creates a report.

The storage unit 530 stores, for example, data of a classifier, which is a learning model used by the above-mentioned recognition unit 5232. The storage unit 530 is realized by the RAM 5002 or the storage 5004 shown in FIG. 11.

The notification unit 531 notifies by displaying various information on the display 5020. For example, the notification unit 531 notifies that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to notifying by display on the display 5020, and may notify by means of an auditory sense such as a voice or an electronic sound such as a warning sound.

The programs for realizing the functions of the information processing device 50 according to the first embodiment are provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD (Compact Disk), a flexible disk (FD), or a DVD (Digital Versatile Disk). Without being limited to this, the programs may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The programs may also be configured to be provided or distributed via a network such as the Internet. The provision or distribution of the above-mentioned programs is similar in the second and third embodiments described below.

The program has a modular configuration including an image acquisition unit 500, a UI unit 501, a control unit 502, an image capture control unit 503, a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, a third state characteristic value calculation unit 523, a report creation unit 524 and an alarm unit 531. In terms of actual hardware, the CPU 5000 reads out the program from a storage medium such as the storage 5004 and executes it, so that the above-mentioned units are loaded onto a main storage device such as the RAM 5002, and the captured image acquisition unit 500, the UI unit 501, the control unit 502, the image capture control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, the first state characteristic value calculation unit 521, the second state characteristic value calculation unit 522, the third state characteristic value calculation unit 523, the report creation unit 524, and the notification unit 531 are generated on the main storage device. Note that the configuration of the above-mentioned program and the manner of generation on the main storage device are the same in the second and third embodiments described below.

(Trigger Generation Method)
Next, a method of generating a trigger for instructing each of the stereo cameras 6L and 6R to capture an image according to the first embodiment will be described in more detail. In the first embodiment, the generating unit 103 generates a trigger at a time interval equal to or shorter than a time period during which the vehicle 1 moves at a speed indicated by the speed information, for a distance in the direction of the speed of the vehicle 1 of the stereo image capturing range of the stereo cameras 6L and 6R with respect to the measured object (road surface 4).

That is, the trigger must be generated so that the imaging ranges of the road surface 4 captured by the stereo cameras 6L and 6R are kept overlapping at a predetermined overlap rate (travel direction overlap rate Dr) or more in the traveling direction of the vehicle 1. As described below, the purpose of this is to be able to detect sufficient corresponding points in order to stably calculate the camera position with high accuracy in the process of calculating the imaging position from each stereo captured image. The lower limit of the traveling direction overlap rate Dr is experimentally determined to be, for example, "60 [%]".

In the first embodiment, the following three methods can be applied to generate a trigger.
(1) Method of generating at regular time intervals (first generation method)
(2) Method of generating by detecting the camera movement speed (second generation method)
(3) Method of calculating and generating a moving distance using a captured image (third generation method)

<First Generation Method>
First, a first generation method of the trigger will be described. In the first generation method, the time interval of the trigger is determined from the maximum speed Speed of the vehicle 1 being imaged and the size of the image capture range (the length of the image capture range in the traveling direction of the vehicle 1). The speed acquisition unit 5021 acquires the maximum speed Speed of the vehicle 1 in advance from the system setting value in the vehicle 1 or a user input to the information processing device 50. The image capture control unit 503 acquires the maximum speed Speed from the speed acquisition unit 5021, and calculates the time interval of the trigger using the acquired maximum speed Speed, the traveling direction field of view Vp, and the traveling direction overlap rate Dr according to the following formula (4). Note that the above-mentioned lower limit value is applied to the traveling direction overlap rate Dr.

The number of triggers to be generated per second (fps) is calculated using formula (4). The inverse of the number of triggers (fps) is the time interval until the next trigger to be generated.

The traveling direction field of view Vp can be set based on the height h of each of the stereo cameras 6L and 6R from the road surface 4 and the angle of view α of each of the stereo cameras 6L and 6R, as explained using FIG. 4. In practice, the traveling direction field of view Vp is set by further taking into consideration the angle of each of the stereo cameras 6L and 6R with respect to the road surface 4, etc.

When the moving vehicle 1 turns right or left, the movement amount of the outer one of the stereo cameras 6L and 6R becomes large. Therefore, rather than using the maximum speed Speed of the vehicle 1 as is, it is more preferable to use the speed of rotational movement based on the position of the outer camera.

The imaging control unit 503 stores and accumulates all stereo images captured in response to the generated trigger, for example, in the storage 5004 or the RAM 5002.

<Second Generation Method>
Next, a second generation method of the trigger will be described. While the above-mentioned first method is simple, when the vehicle 1 is stopped or moving at a speed slower than a predetermined speed, images are captured at intervals that are excessively short for the maximum speed Speed, resulting in a large amount of accumulated stereo captured images. In the second generation method, the image capture control unit 503 detects the moving speed of the camera and generates a trigger according to the detected moving speed.

The imaging control unit 503 uses the current speed of the vehicle 1 indicated by the speed information acquired by the speed acquisition unit 5021 as the maximum speed Speed in equation (4) to calculate the time interval until the next trigger, and performs imaging. According to the second generation method, the slower the moving speed of the vehicle 1, the longer the time interval for trigger generation, and the less likely it is that unnecessary imaging will be performed.

The imaging control unit 503 stores and accumulates all stereo images captured in response to the generated trigger, for example, in the storage 5004 or the RAM 5002.

Note that this second generation method and the first generation method described above can be implemented in combination.

<Third Generation Method>
Next, a third generation method of the trigger will be described. In the third generation method, like the first generation method described above, a trigger is generated at a fixed time interval based on the maximum speed Speed of the vehicle 1. Here, in the third generation method, the stereo captured images captured in response to the trigger are not all stored, but are stored only when the overlap rate between the last stored stereo captured image and the stereo captured image captured immediately before falls below a preset traveling direction overlap rate Dr.

As will be described later, it is possible to calculate the distance traveled by the camera (vehicle 1) using only the stereo images captured so that they overlap in the direction of travel of vehicle 1.

The imaging control unit 503 calculates the movement distance of the camera using the last stored stereo image and the stereo image captured immediately before. The imaging control unit 503 determines whether the calculated distance exceeds the movement distance corresponding to the lower limit of the traveling direction overlap rate Dr. If the imaging control unit 503 determines that the calculated distance exceeds the lower limit, it stores the stereo image captured immediately before. On the other hand, if the imaging control unit 503 determines that the calculated distance does not exceed the lower limit, it discards the stereo image captured immediately before.

This means that even if a sensor is not used to measure the current speed of vehicle 1, unnecessary image accumulation is avoided when the moving speed is slow.

In the above description, the trigger for instructing the stereo cameras 6L and 6R to capture an image is generated outside the stereo cameras 6L and 6R, for example, in the information processing device 50, and transmitted to the stereo cameras 6L and 6R, but this is not limited to the above. For example, the trigger may be generated in one of the stereo cameras 6L and 6R, which instructs the stereo cameras 6L and 6R to capture an image.

(Calculation method of road surface property values)
Next, a method for calculating road surface property values according to the first embodiment will be described below. A method for measuring the flatness σ, the crack rate C, and the rutting depth D according to the first embodiment will be described below.

<Flatness>
13 is a flowchart showing an example of the flatness calculation process according to the first embodiment. In step S100, the captured image acquisition unit 500 acquires stereo captured images captured by the stereo cameras 6L and 6R and stored in, for example, the storage 5004. When the stereo captured images are acquired, the process proceeds to steps S101a and S101b which can be processed in parallel.

In step S101a, a depth map is generated based on the stereoscopic images acquired in step S100. A depth map generation process applicable to the first embodiment will be described with reference to Figs. 14 and 15. Fig. 14 is a flowchart showing an example of the depth map generation process of the first embodiment.

In step S120, the matching processing unit 510 acquires a stereo image from the captured image acquisition unit 500. In the next step S121, the matching processing unit 510 performs matching processing based on the two captured images that make up the acquired stereo image. In the next step S122, the 3D information generation unit 511 calculates depth information based on the matching processing result of step S121, and generates a depth map, which is three-dimensional point cloud information.

The processing of steps S121 and S122 will be described in more detail. In the first embodiment, the depth information is calculated by the stereo method using two captured images that constitute a stereo captured image. The stereo method here is a method that uses two captured images captured from different viewpoints by two cameras, determines a corresponding pixel (corresponding pixel) in one captured image for a pixel (reference pixel) in the other captured image, and calculates the depth (depth value) by trigonometry based on the reference pixel and the corresponding pixel.

In step S121, the matching processing unit 510 uses the two captured images that constitute the stereo captured image acquired from the captured image acquisition unit 500, and performs a search by moving within the other captured image an area to be searched that corresponds to an area of a predetermined size centered on a reference pixel in one of the captured images that serves as a reference.

Various methods are known for searching for corresponding pixels, and for example, block matching can be applied. In the block matching method, pixel values of an area cut out as a block of M pixels by N pixels centered on a reference pixel in one captured image are obtained. In addition, pixel values of an area cut out as a block of M pixels by N pixels centered on a target pixel in the other captured image are obtained. Based on the pixel values, the similarity between the area including the reference pixel and the area including the target pixel is calculated. The similarity is compared while moving the M pixel by N pixel block within the image to be searched, and the target pixel in the block at the position where the similarity is highest is determined to be the corresponding pixel that corresponds to the reference pixel.

The similarity can be calculated by various calculation methods. For example, the normalized cross-correlation (NCC) shown in the following formula (5) is one of the cost functions, and the larger the value of the cost numerical value C _NCC , the higher the similarity. In formula (5), the values M and N represent the size of the pixel block for performing the search. Furthermore, the value I(i, j) represents the pixel value of a pixel in a pixel block in one captured image that is a reference, and the value T(i, j) represents the pixel value in a pixel block in the other captured image that is the search target.

As described above, the matching processing unit 510 performs the calculation of formula (5) while moving a pixel block in one captured image that corresponds to a pixel block of M pixels by N pixels in the other captured image, for example, in pixel units within the other captured image, to calculate the numerical value C _NCC . In the other captured image, the central pixel of the pixel block with the maximum numerical value C _NCC is set as the target pixel corresponding to the reference pixel.

Returning to the explanation of FIG. 14, in step S122, the 3D information generating unit 511 uses trigonometry to calculate a depth value (depth information) based on the reference pixel and the corresponding pixel obtained by the matching process in step S121, and generates three-dimensional point cloud information relating to one captured image and the other captured image that constitute the stereo captured image.

Fig. 15 is a diagram for explaining trigonometry. The purpose of the process is to calculate the distance S to a target object 403 (e.g., a point on the road surface 4) in the figure from imaging position information in an image captured by each imaging element 402 (corresponding to imaging elements _601L and _601R ). In other words, this distance S corresponds to the depth information of the target pixel. The distance S is calculated by the following formula (6).

In addition, in formula (6), the value baseline represents the length of the baseline (baseline length) between the cameras 400a and 400b. In the example of FIG. 5, this corresponds to the baseline length of the imaging lenses 6L _1L and 6L _R (in the case of the stereo camera 6L). The value f represents the focal length of the lens 401 (corresponding to the imaging lenses 6L _1L and 6L _R ). The value q represents the parallax. The parallax q is a value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel by the pixel pitch of the imaging element. The coordinate values of the corresponding pixel are obtained based on the result of the matching process in step S121.

This formula (6) is a calculation method for the distance S when two cameras 400a and 400b, i.e., imaging lenses _{6L 1L} and 6L _1R , are used. This is for calculating the distance S from the captured images captured by the two cameras 400a and 400b, i.e., imaging lenses 6L 1L and 6L _1R . In the first embodiment, the calculation method using this formula (6) is applied to the captured images captured by the imaging lenses _{6L 1L} and 6L _1R of the stereo camera 6L and the imaging lenses 6R _1L and 6R _1R of the stereo camera 6R to calculate the distance S for each pixel.

Returning to the explanation of FIG. 13, in step S101b, the 3D information generator 511 estimates the camera position and orientation. Here, the camera position indicates, for example, the central coordinates of the stereo camera 6L when captured as a single unit. Details of the processing in step S101b will be described later.

When the processing of steps S101a and S101b is completed, the process proceeds to step S102. In step S102, the 3D information generation unit 511 performs coordinate transformation of the depth map of the next time point after the previous time point for two depth maps adjacent in the time series, based on the camera position and orientation estimated in step S101b, so as to match the coordinate system of the depth map of the previous time point.

Note that the previous time and next time are defined as the time when the first imaging was performed and the next time when the second imaging was performed, in which the first and second imaging were performed consecutively in time series in response to a trigger by the stereo cameras 6L and 6R. In other words, the depth map for the previous time is a depth map generated based on the stereo captured images from the first imaging, and the depth map for the next time is a depth map generated based on the stereo captured images from the second imaging.

In the next step S103, the 3D information generation unit 511 integrates the depth map for the next time that has been coordinate-converted in step S102 with the depth map for the previous time. That is, the two depth maps for the previous and next times are aligned in a common coordinate system by the coordinate conversion in step S102, and therefore these two depth maps can be integrated. The processes in steps S102 and S103 are performed on the stereo images captured at all times.

As shown in Figures 5 and 7, when multiple stereo cameras 6L and 6R, or multiple stereo cameras 6L, 6C, and 6R are used lined up in the road width direction, the camera positions and orientations of the stereo cameras lined up in the road width direction (for example, stereo cameras 6L and 6R) are estimated in the same manner, and then depth map integration is performed.

In addition, in the third generation method described above, the movement distance of the camera (vehicle 1) can be calculated in the coordinate conversion in step S102 and the depth map integration process in step S103.

In the next step S104, the 3D information acquisition unit 520 acquires the depth map integrated in step S103 from the 3D information generation unit 511. The second state characteristic value calculation unit 522 then calculates the flatness σ based on the depth map acquired by the 3D information acquisition unit 520. That is, when the depth maps are integrated, the road surface shape of the road in the measured section is restored as a point cloud in a single three-dimensional space. When the point cloud is restored, a coordinate system is taken that connects the coordinates of points 1.5 m forward and backward according to the regulations for each sampling point on the road surface, and the distance to the three-dimensional point cloud in the center is calculated, so that the displacement amount d can be calculated in the same way as in formula (2), and the flatness σ can be calculated from this displacement amount d according to formula (3).

As for the rutting amount D, in step S104, the third state characteristic value calculation unit 523 can obtain the depths _D1 and _D2 as shown in Figures 1(a) and 1(b) by scanning the depth map acquired by the 3D information acquisition unit 520 in the road width direction. The rutting amount D can be calculated based on the acquired depths _D1 and _D2 and the information on the cross section obtained by scanning the depth map. The measurement of the rutting amount D will be described later in detail.

<Crack rate>
The first state characteristic value calculation unit 521 applies the stereo captured images acquired in step S100 to the depth map integrated in step S103, for example. That is, the first state characteristic value calculation unit 521 integrates each of the stereo captured images captured in an overlapping manner with an overlap rate equal to or greater than the traveling direction overlap rate Dr in the traveling direction of the vehicle 1. The first state characteristic value calculation unit 521 sets a 50 cm mesh for the integrated image in accordance with a regulation, acquires information such as cracks and patching in each mesh by image analysis, and calculates a crack rate C in accordance with a regulation based on the acquired information.

<Camera position and orientation estimation process>
Next, the estimation process of the camera position and orientation in step S101b in the flowchart of Fig. 13 described above will be described in more detail. Fig. 16 is a flowchart showing an example of the estimation process of the camera position and orientation in the first embodiment. In the first embodiment, the camera position and orientation are estimated using the above-mentioned SfM.

In step S130, the 3D information generating unit 511 acquires stereo images from the captured image acquiring unit 500. Here, the 3D information generating unit 511 acquires two stereo images captured consecutively in time series (a stereo image at the previous time and a stereo image at the next time). Next, in step S131, the 3D information generating unit 511 extracts feature points from each acquired stereo image. This process of extracting feature points is a process of finding points that are easy to detect as corresponding points in each stereo image, and typically detects points called corners where there is a change in the image and where the change is not uniform.

In the next step S132, the 3D information generation unit 511 detects points in the stereo image captured at the next time that are captured at the same locations as the points extracted as feature points in the stereo image captured at the previous time. This detection process can use a method called optical flow or a method called feature point matching, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Upped Robust Feature).

In the next step S133, the 3D information generation unit 511 estimates the initial position and orientation of the camera. The 3D information generation unit 511 estimates the camera position and calculates three-dimensional coordinates by solving simultaneous equations using the coordinates of the corresponding points detected in each stereo image in step S132 as fixed values and the position and orientation of the camera that captured the next stereo image as parameters (step S134).

A method for estimating the initial position and orientation of a camera will be described with reference to Fig. 17. The following formula (7) represents the relationship between coordinates _xij of a point _Xj in space projected onto a camera (viewpoint) _Pj . In formula (7), the value n represents the number of points in the three-dimensional space, and the value m represents the number of cameras (captured images).

Each value P is a projection matrix that converts the coordinates of a point in three-dimensional space into two-dimensional coordinates of the image for each camera, and is expressed by the following formula (8). Formula (8) is composed of a two-row, two-column transformation matrix of three-dimensional coordinates shown on the right side, and a projective transformation f _i from three-dimensional coordinates to two-dimensional coordinates.

17, the values _Xj and _Pj are calculated using simultaneous equations given only the coordinates _xij captured by the camera. The linear least squares method can be used to solve these simultaneous equations.

In the next step S135, the 3D information generation unit 511 optimizes the three-dimensional coordinates indicating the camera position calculated in step S134. The camera position and orientation calculated by solving the simultaneous equations may not have sufficient accuracy. Therefore, the calculated values are used as initial values to perform optimization processing, thereby improving accuracy.

The difference between the corresponding point coordinates acquired by searching on the image and the two-dimensional coordinates x _ij (called reprojected coordinates) calculated by the above-mentioned formula (7) using the three-dimensional coordinates calculated in step S134 and the projection matrix of formula (8) as parameters is called a residual. The parameters are adjusted by optimization calculation so that the sum of the residuals is minimized for all corresponding points in all stereo captured images used to calculate the camera position and orientation. This is called bundle adjustment. By performing global optimization by bundle adjustment, the accuracy of the camera position and orientation can be improved.

The above is the base process for estimating the camera position and orientation in normal SfM. In the first embodiment, stereo cameras 6L and 6R with known baseline lengths are used, so in addition to detecting corresponding points in stereo images captured with overlapping imaging ranges in accordance with the movement of vehicle 1, it is easy to search for corresponding points in the two captured images that make up the stereo image. Therefore, the three-dimensional coordinates Xj in space are determined at a real scale (actual size), and the position and orientation of the camera after movement can also be stably calculated.

<Amount of rutting>
Next, the measurement process of the rutting amount D in step S104 in the flowchart of Fig. 13 described above will be described in more detail. Fig. 18 is a flowchart showing an example of the rutting amount measurement process of the first embodiment. Fig. 19 is a diagram showing an example of an image showing the foreign object recognition result of the first embodiment. Fig. 20 is a diagram for explaining the measurement position of the rutting amount. Fig. 21 is a diagram showing the measurement position of the rutting amount.

In step S200, the input unit 5231 of the third state characteristic value calculation unit 523 inputs the depth map (three-dimensional point cloud information) acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500 (hereinafter, simply referred to as captured image in FIG. 18).

In the next step S201, the recognition unit 5232 of the third state characteristic value calculation unit 523 uses a classifier, which is a learning model obtained by supervised learning based on learning data about foreign object information on the road surface that has been added in advance, to recognize whether or not a foreign object is included in the subject shown in the captured image input by the input unit 5231. At this time, the recognition unit 5232 reads out the classifier data from the memory unit 530.

In the next step S202, the recognition unit 5232 determines whether or not the recognition process using the classifier has been completed for all captured images input by the input unit 5231. If the process has been completed up to the last captured image (step S202: Yes), the process proceeds to step S203, and if the process has not been completed (step S202: No), the process returns to step S200.

In the next step S203, the recognition unit 5232 outputs the recognition result for the captured image using the classifier to the determination unit 5233. Here, FIG. 19 shows the recognition result of a foreign object for the captured image by the recognition unit 5232 using the classifier. The identification image CI showing the recognition result for the captured image in FIG. 19 is an example of an image showing the recognition result for six labels (no color, manhole, road sign, white line, road shoulder, etc.), and in the identification image CI, the manhole portion 901, the white line portion 902, the road sign portion 903, the road shoulder portion 904, and the other portion 905 are recognized as foreign objects. Conversely, in the identification image CI, areas other than those recognized as foreign objects are recognized as not having any foreign objects.

In the next step S204, the determination unit 5233 of the third state characteristic value calculation unit 523 determines initial measurement positions for measuring the amount of rutting in the captured image input by the input unit 5231. Then, the determination unit 5233 sets measurement sections (predetermined ranges) that are surrounded by a predetermined length in both the vehicle width direction and the traveling direction of the vehicle 1, with each of the determined initial measurement positions being the center in the traveling direction of the vehicle 1.

The determination unit 5233 determines initial measurement positions at appropriate intervals, for example, at five locations within a range of 100 [m] in the traveling direction. For example, as shown in FIG. 20, the determination unit 5233 determines five locations, 10 [m], 30 [m], 50 [m], 70 [m], and 90 [m], as initial measurement positions within a range of 100 [m] in the traveling direction. Then, as shown in FIG. 20, the determination unit 5233 sets a measurement section having a length of 10 [m], 5 [m] forward and backward, with each of the determined initial measurement positions being the center in the traveling direction of the vehicle 1, and a predetermined measurement width defined by a measurement range line in the vehicle width direction of the vehicle 1. In FIG. 20, measurement sections A to E correspond to the initial measurement positions of 10 [m], 30 [m], 50 [m], 70 [m], and 90 [m], respectively. In Figure 20, the direction from measurement section A to measurement section E is the traveling direction of vehicle 1, and this also applies to Figure 21 described below.

In the next step S205, the determination unit 5233 determines whether or not a foreign object has been avoided at a position in the image showing the recognition result (identification image) that corresponds to the measurement position determined in the captured image. If a foreign object has been avoided (step S205: Yes), the process proceeds to step S207, and if a foreign object has not been avoided (step S205: No), the process proceeds to step S206. If a foreign object has not been avoided, the notification unit 531 may notify the user that the rutting amount D cannot be measured at the measurement position.

In step S206, if a foreign object is present at a position in the image showing the recognition result (identification image) that corresponds to the measurement position determined in the captured image, the determination unit 5233 shifts the measurement position by a predetermined length (e.g., a specific number of pixels) forward or backward in the traveling direction of the vehicle 1 and determines it as a new measurement position (i.e., changes the measurement position). For example, as shown in FIG. 21(a), if a foreign object is present at the initial measurement position determined in the measurement section, the determination unit 5233 shifts the measurement position by a predetermined length forward (left side) in the traveling direction of the vehicle 1 and determines it as a new measurement position, as shown in FIG. 21(b). FIG. 21(b) shows that the foreign object was avoided at the new measurement position as a result of shifting the measurement position. After the new measurement position is determined, the process returns to step S205.

Here, the method of shifting the measurement position may be, for example, a method of continuing to shift a predetermined length forward in the traveling direction of the vehicle 1 to a position that avoids foreign objects, a method of continuing to shift a predetermined length backward in the traveling direction of the vehicle 1 to a position that avoids foreign objects, or a method of continuing to shift a predetermined length alternately forward and backward in the traveling direction, and any method may be adopted. Among these, in the case of the method of continuing to shift a predetermined length alternately forward and backward in the traveling direction, the final measurement position can be determined as close to the initial position as possible, and it is possible to suppress the variation of each determined measurement position in relation to other measurement sections, and accurate measurement within a 100 [m] section is possible. Note that this method of shifting the measurement position is the same in the second and third embodiments described later.

In step S206, if the new measurement position determined by the determination unit 5233 falls outside the measurement section, the measurement unit 5234 may not measure the amount of rutting D at that measurement position. In this case, the notification unit 531 may notify the user that the amount of rutting D will not be measured at that measurement position.

In step S207, if no foreign object is present at a position in the image showing the recognition result (identification image) that corresponds to the measurement position determined in the captured image, the determination unit 5233 determines that measurement position as the measurement position for measuring the amount of rutting D. In this way, the determination unit 5233 determines the measurement position for measuring the amount of rutting D in each set measurement section through the processing of steps S205 to S207.

In the next step S208, the measurement unit 5234 of the third state characteristic value calculation unit 523 measures the amount of rutting D at a position on the depth map (three-dimensional point cloud information) corresponding to the measurement position determined by the determination unit 5233. Specifically, as described above, the measurement unit 5234 acquires depths D1 and _D2 as shown in Figures 1(a) and 1(b) by scanning in the road width direction at the determined position on the depth map, and measures the amount of rutting D based on the depths _D1 and _D2 and information on the cross section obtained _by scanning the depth map.

In the next step S209, the measurement unit 5234 stores the measured rutting amount D in the memory unit 530.

As described above, the information processing device 50 according to this embodiment uses a classifier, which is a learning model obtained by supervised learning (deep learning or machine learning) based on learning data about foreign object information on the road surface that has been added in advance, to recognize whether or not a foreign object is included in the subject shown in the captured image. Then, when the information processing device 50 determines that a foreign object is present at the determined measurement position, it sets a position shifted a predetermined length from the measurement position as a new measurement position and further determines whether or not a foreign object is present. In this way, the classifier, which is a learning model, can automatically recognize foreign objects in the captured image, and measurement processing such as the amount of rutting during road inspections can be performed accurately and quickly.

In the example of the identification image CI shown in FIG. 19, examples of foreign objects include road signs and manholes, which can cause erroneous measurement results due to their paint thickness when measuring the amount of rutting D, but are not limited to these. Grass, poles, people, parked vehicles, etc. that are present in the measurement section may also be avoided from the measurement position as foreign objects.

The classifier stored in the storage unit 530 may be updated or newly generated by supervised learning (deep learning or machine learning) using new additional information (learning data) about abnormality information on the road surface. Here, the subject that performs the learning is not shown in the functional block shown in FIG. 12, but may be included in the information processing device 50 as a learning unit that performs the above-mentioned learning, or the data of the classifier (learning model) generated by learning using the learning data by an external device may be stored in the storage unit 530 and used.

As described above, in the imaging system 10 according to the first embodiment, a trigger is generated in the information processing device 50 and distributed to each of the stereo cameras 6L and 6R, so that the stereo cameras 6L and 6R perform imaging in synchronization. At this time, due to the influence of the clock generators in each of the stereo cameras 6L and 6R, the trigger distribution parts for distributing the trigger in the information processing device 50, and differences in the trigger wiring length in the information processing device 50 and each of the stereo cameras 6L and 6R, there is a possibility that a slight deviation will occur in the timing at which each of the stereo cameras 6L and 6R actually receives the trigger and performs imaging. It is preferable to suppress this deviation in imaging timing as much as possible.

In particular, it is desirable to minimize deviations in imaging timing between the stereo cameras 6L and 6R, whose baseline lengths are known. For example, when wiring triggers from the camera I/F 5010a to the stereo cameras 6L and 6R, the wiring may go through a repeater to ensure the quality of the trigger or a photocoupler to protect against electrostatic noise, etc. In this case, it is desirable to share these repeaters and photocouplers between the stereo cameras 6L and 6R.

Second Embodiment
Next, the second embodiment will be described, focusing on the differences from the first embodiment. In the above-mentioned first embodiment, the presence or absence of a foreign object at a measurement position in a captured image (depth map) is recognized using a learning model obtained by supervised learning (deep learning or machine learning) based on learning data on foreign object information on the road surface that has been added in advance. In this embodiment, an operation of determining whether the rutting amount obtained at the measurement position is an abnormal value or not, and changing the measurement position if it is an abnormal value will be described. Note that the configuration of the imaging system 10 according to this embodiment and the hardware configuration of the information processing device are the same as those in the first embodiment.

(Configuration of Functional Blocks of Information Processing Device)
22 is a diagram showing an example of a functional block configuration for explaining functions of an information processing device according to a second embodiment. In FIG. 22, an information processing device 50a (an example of a measurement device) includes an image acquisition unit 500, a UI unit 501, a control unit 502, and an image capture control unit 503. The information processing device 50a further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, a third state characteristic value calculation unit 523a, a report creation unit 524, a storage unit 530a, and a notification unit 531. The operations of the captured image acquisition unit 500, the UI unit 501, the control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, the first state characteristic value calculation unit 521, the second state characteristic value calculation unit 522 and the report creation unit 524 are as described in the first embodiment.

The captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523a, and report creation unit 524 are realized by a program that runs on the CPU 5000. Without being limited to this, some or all of the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523a, and report creation unit 524 may be configured by hardware circuits that operate in cooperation with each other.

The third state characteristic value calculation unit 523a calculates the rutting amount D (an example of a physical quantity) from among the state characteristic values for determining the MCI, using the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. As shown in FIG. 22, the third state characteristic value calculation unit 523a has an input unit 5231a, a determination unit 5233a, a measurement unit 5234a, and an output unit 5235a.

The input unit 5231a inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. The stereo image input by the input unit 5231a may be any one of the stereo images captured by the stereo cameras 6L and 6R.

The determination unit 5233a determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point cloud information (depth map).

The measuring unit 5234a measures the rutting amount D at the measurement position determined by the determining unit 5233a. The measuring unit 5234a then determines whether the rutting amount D was measured normally and whether the measured rutting amount D is an abnormal value.

The output unit 5235a outputs the rutting amount D measured by the measurement unit 5234a that is not an abnormal value to the report creation unit 524.

The memory unit 530a stores, for example, data used when determining whether the amount of rutting measured by the measuring unit 5234a is an abnormal value. The memory unit 530a is realized by the RAM 5002 or the storage 5004 shown in FIG. 11.

The notification unit 531 notifies by displaying various information on the display 5020. For example, the notification unit 531 notifies that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233a, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to notifying by display on the display 5020, and may notify by means of an auditory sense such as a voice or an electronic sound such as a warning sound.

(Method of calculating the amount of rutting among road surface property values)
Next, the process according to this embodiment will be described in more detail with respect to the process of measuring the rutting amount D in step S104 in the flowchart of Fig. 13. Fig. 23 is a flowchart showing an example of the process of measuring the rutting amount according to the second embodiment.

In step S210, the input unit 5231a of the third state characteristic value calculation unit 523a inputs the depth map (three-dimensional point cloud information) acquired by the 3D information acquisition unit 520 and each stereo captured image (hereinafter simply referred to as captured image in FIG. 23) acquired by the captured image acquisition unit 500. The determination unit 5233a of the third state characteristic value calculation unit 523a then determines an initial measurement position for measuring the amount of rutting in the captured image input by the input unit 5231a. The determination unit 5233a then sets a measurement section that is surrounded by a predetermined length in the vehicle width direction and the traveling direction of the vehicle 1, with each determined initial measurement position being the center in the traveling direction of the vehicle 1. The specific measurement section setting process is as described in the first embodiment.

In the next step S211, the measurement unit 5234a of the third state characteristic value calculation unit 523a measures the amount of rutting D at a position in the depth map (three-dimensional point cloud information) that corresponds to the measurement position determined by the determination unit 5233a.

In the next step S212, the measuring unit 5234a judges whether the rutting amount D was measured normally. For example, the rutting amount D cannot be measured normally when the part between the two "ruts" cannot be detected and the depths _D1 and _D2 cannot be obtained, as shown in Fig. 1(a) and Fig. 1(b). If the rutting amount D was measured normally (step S212: Yes), the process proceeds to step S213, and if the rutting amount D was not measured normally (step S212: No), the process proceeds to step S214. If the rutting amount D cannot be measured normally, the notification unit 531 may notify the user that the rutting amount D cannot be measured normally at the measurement position.

In step S213, if the rutting amount D is measured normally, the measuring unit 5234a further determines whether the rutting amount D is an abnormal value. For example, the measuring unit 5234a may determine whether the measured rutting amount D is an abnormal value by comparing the measured rutting amount D with a previously measured rutting amount D or an average rutting amount, or by determining whether the measured rutting amount D is within a predetermined normal range. In this case, data on the previously measured rutting amount D, the average rutting amount, the predetermined normal range, etc. may be stored in the storage unit 530a in advance. If the rutting amount D is not an abnormal value (step S213: Yes), the process proceeds to step S215. If the rutting amount D is an abnormal value (step S213: No), the process proceeds to step S214.

In step S214, if the rutting amount D measured by the measurement unit 5234a cannot be measured normally or is an abnormal value, the determination unit 5233a shifts the measurement position forward or backward in the traveling direction of the vehicle 1 by a predetermined length (e.g., a specific number of pixels) and determines it as a new measurement position (i.e., changes the measurement position). The specific method for determining the new measurement value is as described in the first embodiment. After the new measurement position is determined, the process returns to step S211.

In step S214, if the new measurement position determined by the determination unit 5233a falls outside the measurement section, the measurement unit 5234a may not measure the rutting amount D at that measurement position. In this case, the notification unit 531 may notify the user that the rutting amount D will not be measured at that measurement position.

In step S215, the measurement unit 5234a determines the measured rutting amount D as a normally measured value. The determination unit 5233a also determines the measurement position where the rutting amount D was normally measured by the measurement unit 5234a as the final (normal) measurement position.

In the next step S216, the measurement unit 5234a stores the determined rutting amount D in the memory unit 530a.

As described above, the information processing device 50a according to this embodiment determines whether the rutting amount D measured at the determined measurement position was measured normally and whether it is an abnormal value. If the measured rutting amount D was not measured normally or is an abnormal value, the information processing device 50a measures the rutting amount D again at a position shifted a predetermined length from the current measurement position as the new measurement position. This makes it possible to automatically recognize whether the measured rutting amount D was measured normally, and allows measurement processing of the rutting amount, etc. during road inspections to be performed accurately and quickly.

[Third embodiment]
Next, the third embodiment will be described, focusing on the differences from the first embodiment. In this embodiment, the operation of determining whether or not a foreign object is present at a measurement position by comparing anomaly information acquired from an anomaly information database (hereinafter referred to as an anomaly information DB) with a determined measurement position, and changing the measurement position if a foreign object is present, will be described. Note that the configuration of the imaging system 10 according to this embodiment and the hardware configuration of the information processing device are similar to those in the first embodiment.

(Configuration of Functional Blocks of Information Processing Device)
24 is a diagram showing an example of a functional block configuration for explaining the function of the information processing device according to the third embodiment. In FIG. 24, the information processing device 50b (an example of a measuring device) includes an image acquisition unit 500, a UI unit 501, a control unit 502, and an image capture control unit 503. The information processing device 50b further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a first state characteristic value calculation unit 521, a second state characteristic value calculation unit 522, a third state characteristic value calculation unit 523b, a record creation unit 524, a storage unit 530b, and a notification unit 531. The operations of the captured image acquisition unit 500, the UI unit 501, the control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, the first state characteristic value calculation unit 521, the second state characteristic value calculation unit 522 and the report creation unit 524 are as described in the first embodiment.

The captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523b, and report creation unit 524 are realized by a program that runs on the CPU 5000. Without being limited to this, some or all of the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, first state characteristic value calculation unit 521, second state characteristic value calculation unit 522, third state characteristic value calculation unit 523b, and report creation unit 524 may be configured by hardware circuits that operate in cooperation with each other.

The third state characteristic value calculation unit 523b calculates the rutting amount D (an example of a physical quantity) from among the state characteristic values for determining the MCI, using the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. As shown in FIG. 24, the third state characteristic value calculation unit 523b has an input unit 5231b, a determination unit 5233b, a measurement unit 5234b, and an output unit 5235b.

The input unit 5231b inputs the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and each stereo image acquired by the captured image acquisition unit 500. The stereo image input by the input unit 5231b may be any one of the stereo images captured by the stereo cameras 6L and 6R.

The determination unit 5233b determines the measurement position of the rutting amount D in the traveling direction of the vehicle 1 on the three-dimensional point cloud information (depth map).

The measurement unit 5234b measures the amount of rutting D at the measurement position determined by the determination unit 5233b.

The output unit 5235b outputs the rutting amount D measured by the measurement unit 5234b to the report creation unit 524.

The memory unit 530b stores a foreign object information DB that accumulates foreign object information, which is data identifying the location of a foreign object by visual inspection or the like, from the past. The memory unit 530b is realized by the RAM 5002 or storage 5004 shown in FIG. 11.

The notification unit 531 notifies by displaying various information on the display 5020. For example, the notification unit 531 notifies that the rutting amount cannot be measured at the measurement position determined by the determination unit 5233b, or that the final measurement position has not yet been determined. Note that the notification unit 531 is not limited to notifying by display on the display 5020, and may notify by means of an auditory sense such as a voice or an electronic sound such as a warning sound.

(Method of calculating the amount of rutting among road surface property values)
Next, the process of measuring the rutting amount D in step S104 in the flowchart of Fig. 13 will be described in more detail according to this embodiment. Fig. 25 is a flowchart showing an example of the process of measuring the rutting amount according to the third embodiment.

In step S220, the input unit 5231b of the third state characteristic value calculation unit 523b inputs the depth map (three-dimensional point cloud information) acquired by the 3D information acquisition unit 520 and each stereo captured image (hereinafter simply referred to as captured image in FIG. 25) acquired by the captured image acquisition unit 500. The determination unit 5233b of the third state characteristic value calculation unit 523b then determines an initial measurement position for measuring the amount of rutting in the captured image input by the input unit 5231b. The determination unit 5233b then sets a measurement section that is surrounded by a predetermined length in the vehicle width direction and the traveling direction of the vehicle 1, with each of the determined initial measurement positions being the center in the traveling direction of the vehicle 1. The specific measurement section setting process is as described in the first embodiment.

In the next step S221, the input unit 5231b further extracts and inputs anomaly information corresponding to the input captured image from the anomaly information DB stored in the memory unit 530b.

In the next step S222, the determination unit 5233b determines whether or not a foreign object has been avoided at the determined measurement position based on the comparison result between the captured image input by the input unit 5231b and the abnormality information extracted from the abnormality information DB. If a foreign object has been avoided (step S222: Yes), the process proceeds to step S224, and if a foreign object has not been avoided (step S222: No), the process proceeds to step S223. If a foreign object has not been avoided, the notification unit 531 may notify the user that the rutting amount D cannot be measured at the measurement position.

In step S223, if a foreign object is present at the measurement position determined in the captured image, the determination unit 5233b shifts the measurement position forward or backward in the traveling direction of the vehicle 1 by a predetermined length (e.g., a specific number of pixels) and determines it as a new measurement position (i.e., changes the measurement position). The specific method for determining the new measurement value is as described in the first embodiment. After determining the new measurement position, the process returns to step S222.

In addition, in step S223, if the new measurement position determined by the determination unit 5233b is outside the measurement section, the measurement unit 5234b may not measure the rutting amount D at that measurement position. In this case, the notification unit 531 may notify that the rutting amount D will not be measured at that measurement position.

In step S224, if no foreign object is present at the measurement position determined in the captured image, the determination unit 5233b determines the measurement position as the correct measurement position for measuring the amount of rutting D. In this way, the determination unit 5233b determines the measurement position for measuring the amount of rutting D in each set measurement section through the processing of steps S222 to S224.

In the next step S225, the measurement unit 5234b of the third state characteristic value calculation unit 523b measures the amount of rutting D at a position on the depth map (three-dimensional point cloud information) corresponding to the measurement position determined by the determination unit 5233b. Specifically, as described above, the measurement unit 5234b acquires depths _D1 and _D2 as shown in Figures 1(a) and 1(b) by scanning in the road width direction at the determined position on the depth map, and measures the amount of rutting D based on the depths _D1 and _D2 and information on the cross section obtained by scanning the depth map.

In the next step S226, the measurement unit 5234b stores the measured rutting amount D in the memory unit 530b.

As described above, the information processing device 50b according to this embodiment extracts anomaly information corresponding to the captured image from a foreign object information DB that has previously stored foreign object information, which is data identifying the location of a foreign object by visual inspection or the like, and determines whether or not a foreign object has been avoided at the determined measurement position based on the anomaly information. If the information processing device 50b determines that a foreign object is present at the determined measurement position, it sets a position shifted a predetermined length from the measurement position as a new measurement position and further determines whether or not a foreign object is present. This makes it possible to automatically recognize foreign objects in the captured image using the anomaly information stored in the anomaly information DB, and to perform measurement processing such as the amount of rutting during road inspections accurately and quickly.

1 Vehicle 2 Mounting portion 3 Mounting member 4 Road surface 5 PC
6, 6C, 6L, 6R Stereo camera 6C _L , 6C _R imaging lens 6L _L , 6L _R imaging lens 6R _L , 6R _R imaging lens 10 Imaging system 50, 50a, 50b Information processing device 60C, 60L, 60R Stereo imaging range 60L', 60R' Stereo imaging range 60C _L , 60C _R imaging range 60L _L , 60L _R imaging range 60L _L ', 60L _R ' Imaging range 60R _L , 60R _R imaging range 61 Region 100 ₁ , 100 ₂ Imaging section 101 ₁ , 101 ₂ Imaging control section 102 Speed acquisition section 103 Generation section 400a, 400b Camera 401 Lens 402 Imaging element 403 Target object 500 Image acquisition section 501 UI unit 502 Control unit 503 Imaging control unit 510 Matching processing unit 511 3D information generation unit 520 3D information acquisition unit 521 First state characteristic value calculation unit 522 Second state characteristic value calculation unit 523, 523a, 523b Third state characteristic value calculation unit 524 Record creation unit 530, 530a, 530b Memory unit 531 Notification unit _600L , _600R Imaging optical system _601L , _601R Imaging element _602L , _602R Driving unit _603L , _603R Signal processing unit 604 Output unit 901 Manhole unit 902 White line unit 903 Road sign unit 904 Road shoulder unit 905 Others 5000 CPU
5001 ROM
5002 RAM
5003 Graphics I/F
5004 Storage 5005 Input device 5006 Data I/F
5007 Communication I/F
5010 Camera I/F
5020 Display 5021 Speed acquisition unit 5231, 5231a, 5231b Input unit 5232 Recognition unit 5233, 5233a, 5233b Determination unit 5234, 5234a, 5234b Measurement unit 5235, 5235a, 5235b Output unit CI Identification image

JP 2015-197804 A

Claims (5)

a determination unit that determines a measurement position for measuring a predetermined physical quantity of the measurement object based on an image captured by the imaging unit;
a measurement unit that measures the physical quantity of the object to be measured at the measurement position determined by the determination unit;
Equipped with
when the physical quantity measured by the measurement unit is not abnormal, the determination unit determines the measurement position at which the physical quantity was measured as a normal measurement position ; and
The measurement unit does not measure the physical quantity of the object to be measured when the measurement position determined by the determination unit is outside a predetermined range . The measurement device according to claim 1, wherein the determination unit changes the measurement position and determines a new measurement position when the physical quantity measured by the measurement unit is abnormal. 3. The measurement device according to claim 1, further comprising a notification unit that notifies the measurement unit that the physical quantity for the object to be measured will not be measured when the measurement position determined by the determination unit is outside a predetermined range. an imaging unit that captures an image of the object to be measured and obtains a captured image;
a determination unit that determines a measurement position for measuring a predetermined physical quantity of a measurement object based on the captured image;
a measurement unit that measures the physical quantity of the object to be measured at the measurement position determined by the determination unit;
having
when the physical quantity measured by the measurement unit is not abnormal, the determination unit determines the measurement position at which the physical quantity was measured as a normal measurement position ; and
The measurement system is configured such that, when the measurement position determined by the determination unit is outside a predetermined range, the measurement unit does not measure the physical quantity of the object to be measured . The measuring device according to any one of claims 1 to 3 ,
The imaging unit that captures the captured image;
Equipped with
The imaging unit of the vehicle is a plurality of stereo cameras.

Images