JP2019164018A - Imaging system, imaging method, mobile body with imaging system installed, imaging apparatus, and mobile body with imaging apparatus installed - Google Patents

Abstract

To provide an imaging system, an imaging method, a mobile body with an imaging system installed, an imaging apparatus and a mobile body with an imaging apparatus installed, enabling easy obtainment of an image with a sufficient contrast appropriate for measuring characteristics of a road surface when imaging the road surface while driving by a vehicle.SOLUTION: A mobile body 1 includes a first imaging apparatus 2 and a second imaging apparatus 7 installed. The first imaging apparatus is arranged in the mobile body so as to take an image in a vertical direction with regard to a road surface 4 where the mobile body is moving. The second imaging apparatus is arranged so as to take an image of a road surface at a side to which the mobile body is progressing, with regard to the first imaging apparatus. An imaging condition of the first imaging apparatus is determined according to an image taken by the second imaging apparatus.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging system, an imaging method, a moving body equipped with the imaging system, an imaging device, and a moving body equipped with the imaging device.

  On various roads, road surface properties such as the cracking rate of the road surface, the amount of rutting, and flatness are regularly measured in order to ensure vehicle driving safety and comfort, and to extract repair points. When performing these measurements, it has been common in the past to perform vehicle traffic control on the target route and perform measurement manually within the restricted area. In recent years, a technique has been proposed in which road surface information is measured without restricting traffic by acquiring road surface information while traveling with a vehicle equipped with a camera or sensor.

  Patent Document 1 discloses a road correction surface inspection device that captures a road surface using a plurality of illuminations and increases the detection rate of defects such as cracks on the road surface.

  When a camera is mounted on a vehicle and the road surface is imaged while traveling, when detecting cracks or measuring rutting, the image of the feature in the captured image has sufficient contrast. It is desirable. However, sufficient contrast may not be obtained in the captured image depending on the road surface condition, for example, the difference in pavement type and the presence or absence of shadows. In this case, measurement based on the captured image may be difficult.

  In order to solve this problem, there is a method of acquiring an image with an appropriate contrast by illuminating the region to be imaged. However, this method requires a large illumination and a large-capacity battery or generator in order to obtain the amount of light necessary for imaging, which increases costs and requires a dedicated vehicle. There was a point. It is also conceivable to perform automatic exposure according to changes in road surface brightness by general exposure control. However, in the road surface imaging while the vehicle is running, there is a problem that a large number of frames in which sufficient contrast cannot be obtained due to the tracking time lag of automatic exposure may occur.

  The present invention has been made in view of the above, and an object of the present invention is to make it possible to easily acquire an image with sufficient contrast suitable for measurement of road surface properties when imaging a road surface while traveling with a vehicle. And

  In order to solve the above-described problems and achieve the object, the present invention provides a first imaging device that is mounted on a moving body and images a road surface on which the moving body moves, and a first imaging device that is mounted on the moving body. A second imaging device that images the road surface of the moving body in the traveling direction with respect to the device; an imaging control unit that determines imaging conditions of the first imaging device based on a captured image captured by the second imaging device; .

  Advantageous Effects of Invention According to the present invention, when a road surface is imaged while traveling with a vehicle, it is possible to easily obtain an image with sufficient contrast suitable for measurement of road surface properties.

FIG. 1 is a diagram for explaining the measurement of rutting amount. FIG. 2 is a diagram for explaining the measurement of flatness. FIG. 3 is a diagram illustrating a configuration example of the imaging system according to the first embodiment. FIG. 4 is a diagram for explaining an imaging range in the traveling direction of the vehicle using a stereo camera, which can be applied to the first embodiment. FIG. 5 is a diagram for explaining an imaging range in the road width direction of a vehicle by a stereo camera, which can be applied to the first embodiment. FIG. 6 is a diagram illustrating overlap in the traveling direction of the vehicle in the stereo imaging range applied by the stereo camera, applicable to the first embodiment. FIG. 7 is a diagram illustrating an example of an imaging system including three stereo cameras according to the first embodiment. FIG. 8 is a block diagram illustrating an example of a schematic configuration of the imaging system according to the first embodiment. FIG. 9 is a block diagram illustrating an example of a hardware configuration of the imaging system according to the first embodiment. FIG. 10 is a block diagram illustrating a configuration of an example of the stereo camera according to the first embodiment. FIG. 11 is a block diagram illustrating a configuration example of an information processing apparatus applicable to the first embodiment. FIG. 12 is a functional block diagram of an example for explaining the functions of the information processing apparatus applicable to the first embodiment. FIG. 13 is a flowchart illustrating an example of flatness calculation processing according to the first embodiment. FIG. 14 is a flowchart illustrating an example of a depth map generation process applicable to the first embodiment. FIG. 15 is a diagram for explaining trigonometry applicable to the first embodiment. FIG. 16 is a flowchart illustrating an example of camera position and orientation estimation processing that can be applied to the first embodiment. FIG. 17 is a diagram for explaining camera position and orientation estimation processing applicable to the first embodiment. FIG. 18 is a diagram illustrating an example of a location where there is a large luminance change on the road surface. FIG. 19 is a flowchart illustrating an example of processing related to the moving direction camera according to the first embodiment. FIG. 20 is a flowchart illustrating an example of processing related to the road surface imaging camera according to the first embodiment. FIG. 21 is a flowchart of an example illustrating processing related to the moving direction camera according to the first modification of the first embodiment. FIG. 22 is a diagram for explaining a relationship between an imaging range by a traveling direction camera and a stereo imaging range by a road surface imaging camera that can be applied to the first modification of the first embodiment. FIG. 23 is a diagram illustrating an example of an image captured by a road surface imaging camera and a traveling direction camera that can be applied to the first modification of the first embodiment. FIG. 24 is a flowchart of an example illustrating processing related to a road surface imaging camera according to a first modification of the first embodiment. FIG. 25 is a flowchart illustrating an example of processing related to the moving direction camera according to the second modification of the first embodiment. FIG. 26 is a flowchart illustrating an example of imaging condition setting processing according to the second embodiment.

  Exemplary embodiments of an imaging system, an imaging method, a moving body equipped with the imaging system, an imaging device, and a moving body equipped with the imaging device will be described in detail below with reference to the accompanying drawings.

[Outline of existing road property inspection method]
Prior to the description of the embodiment, an existing road property inspection method will be schematically described. As an index for evaluating road properties, a maintenance control index (MCI) is defined. The MCI quantitatively evaluates the pavement serviceability by three kinds of road surface property values of “cracking rate”, “wad digging amount” and “flatness”.

  Of these, the “crack rate” is calculated as follows in each area where the road surface is divided into 50 cm meshes according to the number of cracks and the patching area. Apply to determine crack rate.

Two crack one ⇒0.15M ² Hibihibiware ⇒0.25M ² Nohibi patching area 0 to 25% ⇒ cracks 0 m ²
Patching area 25-75% ⇒Crack 0.125m ²
Patching area 75% or more ⇒ Crack 0.25m ²

As shown in FIG. 1A and FIG. 1B, the “wadder digging amount” was measured by measuring the depths D ₁ and D ₂ of _two “wadachi” generated in one lane. A larger value is adopted among the depths D ₁ and D ₂ . There are several patterns for digging "Wadachi", so a measurement method is defined for each pattern. FIG. 1A shows a case where the distance between two “wadachi” is higher than both ends of the two “wadachi”, and FIG. The example of the measurement method when it is lower than both ends of "Wadachi" is shown.

The flatness is measured at three points at 1.5 m intervals from the reference plane along the traveling direction of the vehicle on the road surface. For example, as shown in FIG. 2, three measuring instruments are provided at 1.5 m intervals on the lower surface of the vehicle, and the heights X ₁ , X ₂ and X ₃ are measured. Based on the measured heights X ₁ , X _2, and X ₃ , the displacement amount d is obtained by Expression (2). This measurement is performed a plurality of times while moving the vehicle. Formula (3) is calculated based on the plurality of displacement amounts d obtained in this way, and flatness σ is calculated.

  The above-described measurement of “cracking rate”, “wad digging amount” and “flatness” is executed for each evaluation section of 100 m. When the damaged part is known in advance, the measurement may be performed by dividing 100 m into smaller units such as 40 m + 60 m. The MCI is calculated based on the measurement result executed in units of 100 m, and a record is created.

  MCI is calculated based on the measured crack rate C [%], rutting amount D [mm], and flatness σ [mm], and the four equations shown in Table 1 are calculated to obtain the values MCI, MCI1, MCI2 and MCI3 is calculated. Of the calculated values MCI, MCI1, MCI2, and MCI3, the minimum value is adopted as the MCI. Evaluation is performed according to the evaluation criteria shown in Table 2 based on the adopted MCI, and it is determined whether or not the road surface of the evaluation section needs repair.

[First Embodiment]
Next, the imaging system according to the first embodiment will be described. In 1st Embodiment, the camera (stereo camera) in which a stereo imaging is possible is attached to a vehicle, and a road surface is imaged. Based on the captured stereo image, depth information with respect to the road surface is acquired from the imaging position, a three-dimensional shape of the road surface is generated, and three-dimensional road surface data is created. By analyzing this three-dimensional road surface data, it is possible to obtain “cracking ratio”, “wedge rubbing amount” and “flatness” used for obtaining MCI.

  This will be described more specifically. The stereo camera includes two cameras provided with a predetermined length (referred to as a baseline length) apart from each other, and outputs two pairs of captured images (referred to as stereo captured images) captured by the two cameras. . By searching for a corresponding point between two captured images included in this stereo captured image, the depth distance can be restored for any point in the captured image. Data in which the depth is restored for the entire captured image and each pixel is represented by depth information is referred to as a depth map. That is, the depth map is three-dimensional point group information including a set of points each having three-dimensional information.

  This stereo camera is attached to one place such as the rear of the vehicle so that the ground can be imaged, and the vehicle is moved along the road to be measured. For the sake of explanation, it is assumed that an imaging range of a stereo camera mounted on a vehicle for measurement covers a specified length in the road width direction.

The rutting amount D arranges the depth values of the portions cut out in a stripe shape in the road width direction in the depth map restored from the stereo captured image captured by the stereo camera. Based on the change of the depth in the road width direction, the rutting depths D ₁ and D ₂ can be calculated.

  The crack rate C is obtained by analyzing a picked-up image obtained by picking up an image of the road surface, detecting “cracks” and the like, and performing the calculation according to the above formula (1) based on the detection result.

  Here, in the traveling direction of the vehicle, since the imaging range by one imaging is limited, for example, the crack rate C in the 100 m section cannot be calculated based on the imaging image by one imaging. Therefore, imaging is sequentially performed every movement according to the length of the vehicle in the imaging range in the imaging range while moving the vehicle along the road. At this time, the imaging trigger is controlled so that the imaging range in the previous imaging and the imaging range in the current imaging overlap at an overlap rate designed in advance or higher.

  In this way, by controlling the imaging trigger according to the movement of the vehicle, the road surface of the road to be measured can be imaged without omission. Therefore, the picked-up images sequentially picked up according to the progress of the vehicle are connected using image processing called stitching to generate one image including, for example, a road surface image of a 100 m section. By visually confirming or analyzing this image, the crack rate C on the road surface can be measured.

  The flatness measurement uses a technique called Structure from Motion (hereinafter referred to as “SfM”) that estimates an imaging position based on images that are sufficiently overlapped from images at different imaging points.

  An overview of SfM processing will be described. First, using images captured with overlapping imaging ranges, a point at which the same point is captured in each image is detected as a corresponding point. It is desirable to detect as many corresponding points as possible. Next, for example, the movement of the camera from the imaging point of the first image to the imaging point of the second image is set as a simultaneous equation using the coordinates of the detected corresponding points using rotation and translation as unknown parameters. Then, a parameter that minimizes the total error is obtained. In this way, the imaging position of the second image can be calculated.

  As described above, the depth of an arbitrary point in the image can be restored as a depth map from the stereo captured image at each imaging point. For the depth map corresponding to the first stereo image, the depth map corresponding to the second stereo image is the depth with the imaging position of the second camera obtained from the above simultaneous equations as the origin. Convert coordinates to a map. Thus, the two depth maps can be unified with the coordinate system of the first depth map. In other words, a single depth map can be generated by combining two depth maps.

  By performing this processing on, for example, depth maps based on all stereo images captured in the 100 m section, the road surface of the 100 m section is restored in one three-dimensional space. The displacement amount d can be calculated by applying the depth value of the road surface restored in this way to the above equation (2). The flatness σ is calculated by applying the displacement amount d to the above equation (3).

  In the first embodiment, road surface flatness σ, rutting amount D for obtaining MCI by performing imaging with a stereo camera mounted on a vehicle and performing image processing on the captured stereo image, and The crack rate C can be measured collectively. In the first embodiment, this measurement can be performed using an imaging system including a stereo camera and an information processing apparatus that performs image processing on a stereo captured image, and MCI can be obtained with a simple configuration. .

[Outline of First Embodiment]
In the first embodiment, when the road surface is picked up by the stereo camera for generating the three-dimensional shape of the road surface, the road surface should be picked up by the stereo camera ahead of the vehicle in which the image pickup system is mounted. Image the range. Based on the captured image captured in the forward imaging range, the imaging condition including the exposure amount when capturing the imaging range with the stereo camera is obtained, and the obtained imaging condition is determined for the stereo camera. Set. Thereby, it is possible to appropriately control the imaging conditions when the road surface is imaged by the stereo camera, and to generate a three-dimensional shape with higher accuracy based on the stereo captured image.

[Camera arrangement applicable to the first embodiment]
Next, an example of camera arrangement applicable to the first embodiment will be described. FIG. 3 is a diagram illustrating a configuration example of the imaging system according to the first embodiment. FIG. 3A is a diagram illustrating a state in which the imaging system according to the first embodiment is mounted on the vehicle 1 from the side surface of the vehicle 1. In FIG. 3A, the direction toward the left end of the figure is the traveling direction of the vehicle 1. That is, in FIG. 3A, the left end side of the vehicle 1 is the front portion of the vehicle 1, and the right end side is the rear portion of the vehicle 1. FIG. 3B is a diagram illustrating an example in which the vehicle 1 is viewed from the rear side.

  In the imaging system according to the first embodiment, an attachment member 3 including an attachment unit 2 for attaching an imaging device is fixed to the rear part of a vehicle 1 as a moving body on which the imaging system is mounted. A stereo camera 6 is attached. Here, as illustrated in FIG. 3B, two stereo cameras 6 </ b> L and 6 </ b> R are attached to both ends in the width direction of the vehicle body of the vehicle 1. Each of the stereo cameras 6L and 6R is attached in a direction for imaging the road surface 4 on which the vehicle 1 moves. Preferably, each stereo camera 6L and 6R is attached so as to image the road surface 4 from the vertical direction.

  Hereinafter, when it is not necessary to distinguish between the stereo cameras 6L and 6R, the stereo cameras 6L and 6R are collectively described as the stereo camera 6.

  The stereo camera 6 is controlled by, for example, a personal computer (PC) 5 installed inside the vehicle 1. The operator operates the PC 5 and instructs the stereo camera 6 to start imaging. When the start of imaging is instructed, the PC 5 starts imaging with the stereo camera 6. The imaging is repeatedly executed with timing controlled according to the moving speed of the stereo camera 6, that is, the vehicle 1.

  For example, the operator operates the PC 5 in response to the end of imaging in a necessary section and instructs the end of imaging. The PC 5 ends the imaging by the stereo camera 6 in response to the instruction to end the imaging.

  For the vehicle 1, a camera 7 is further provided for imaging the road surface 4 ahead of the vehicle 1 in the traveling direction. That is, the camera 7 images a predetermined area of the road surface 4 to be imaged by the stereo camera 6 prior to the imaging of the predetermined area by the stereo camera 6. A captured image captured by the camera 7 (referred to as a traveling direction captured image) is sent to the PC 5. The PC 5 detects the brightness of the predetermined area based on the captured image in the traveling direction sent from the camera 7 and sets the imaging condition including the exposure amount of the stereo camera 6 based on the detection result.

  Hereinafter, the camera 7 is referred to as a traveling direction camera 7. Further, since the stereo cameras 6L and 6R are cameras that capture the road surface 4 from the vertical direction for MCI calculation, unless otherwise specified, the stereo cameras 6L and 6R are described as road surface imaging cameras 6L and 6R. This is distinguished from the moving direction camera 7.

  4 and 5 are diagrams illustrating examples of the imaging range of the road surface imaging camera 6 that can be applied to the first embodiment. 4 and 5, the same reference numerals are given to the portions common to FIG. 3 described above, and the detailed description thereof is omitted.

  FIG. 4 is a diagram for explaining an imaging range in the traveling direction of the vehicle 1 (referred to as a traveling direction visual field Vp) by the road surface imaging camera 6 that can be applied to the first embodiment. Note that, in FIG. 4, the direction toward the left end of the figure is the traveling direction of the vehicle 1, as in FIG. 1 described above. As shown in FIG. 4, the traveling direction visual field Vp is determined according to the angle of view α of the road surface imaging camera 6 and the height h of the road surface imaging camera 6 with respect to the road surface 4.

  FIG. 5 is a diagram for describing an imaging range in the road width direction of the vehicle 1 by the road surface imaging camera 6 applicable to the first embodiment. FIG. 5 is a view of the vehicle 1 as seen from the rear side, and the same reference numerals are given to the parts common to FIG.

In FIG. 5 (a), a road surface imaging camera 6L is provided with two imaging lens 6L _L and 6L _R. A line connecting the imaging lenses 6L _L and 6L _R is referred to as a base line, and the length thereof is referred to as a base line length. The road surface imaging camera 6L is disposed so that the base line is perpendicular to the traveling direction of the vehicle 1. Similarly, the road surface imaging camera 6 </ b> _R includes two imaging lenses 6 </ b> _{R L} and 6 </ b> _{R R} separated by the base line length, and is arranged so that the base line is perpendicular to the traveling direction of the vehicle 1.

FIG. 5B shows an example of the imaging range of the road surface imaging cameras 6L and 6R applicable to the first embodiment. In the road surface imaging camera 6L, the imaging ranges 60L _L and 60L _R of the imaging lenses 6L _L and 6L _R are shifted and overlapped according to the base line length and the height h. Similarly, for the road surface imaging camera 6R, the imaging ranges 60R _L and 60R _R by the imaging lenses 6R _L and 6R _R are shifted and overlapped according to the base line length and the height h.

Hereinafter, unless otherwise specified, the imaging ranges 60L _L and 60L _R and the imaging ranges 60R _L and 60R _R are collectively referred to as the stereo imaging ranges 60L and 60R, respectively. As shown in FIG. 5 (b), the road surface imaging cameras 6L and 6R have a stereo imaging range 60L and 60R, one end of the stereo imaging range 60L in the width direction of the vehicle 1, and the width direction of the stereo imaging range 60R. Are arranged so as to overlap at a predetermined overlap rate in the region 61.

FIG. 6 is a diagram illustrating overlap in the traveling direction of the vehicle 1 of the stereo imaging range 60L by the road surface imaging camera 6L that can be applied to the first embodiment. Assume that in the moving vehicle 1, imaging is performed twice by the road surface imaging camera 6 </ b> L. The road surface imaging camera 6L first images the stereo imaging range 60L (imaging ranges 60L _L and 60L _R ), and the second time, the vehicle 1 in the traveling direction of the vehicle 1 with respect to the stereo imaging range 60L. The stereo imaging range 60L ′ (imaging ranges 60L _L ′ and 60L _R ′) moved by a distance corresponding to the moving distance is captured.

  At this time, the imaging timing of the road surface imaging camera 6L is such that the stereo imaging range 60L and the stereo imaging range 60L ′ overlap with the overlapping rate that is greater than or equal to the traveling direction overlap rate Dr with respect to the traveling direction visual field Vp. To control. In this way, by sequentially capturing the stereo imaging ranges 60L and 60L ′ along the time axis, it is possible to facilitate the process of joining the two stereo captured images obtained by capturing the stereo imaging ranges 60L and 60L ′. .

  In the above description, the imaging system according to the first embodiment has been described as using two road surface imaging cameras 6L and 6R. However, this is not limited to this example. For example, in the imaging system according to the first embodiment, as shown in FIG. 7A, one more stereo camera (road surface imaging camera) 6C is added to the road surface imaging cameras 6L and 6R. You may comprise using the three road surface imaging cameras 6L, 6R, and 6C.

In the example of FIG. 7A, the distance between the road surface imaging cameras 6L and 6R is wider than when only the road surface imaging cameras 6L and 6R are used, and the road surface imaging camera 6C is arranged at the center. Yes. As shown in FIG. 7 (b), the imaging range 60C _L and 60C _R by the image pickup lens 6C _L and 6C _R road imaging camera 6C, stereo imaging range 60C is formed. The road surface imaging cameras 6L, 6C, and 6R are arranged such that the respective stereo imaging ranges 60L, 60C, and 60R overlap in the width direction of the vehicle 1 with a predetermined overlap rate.

  Thus, by using the three road surface imaging cameras 6L, 6C and 6R to image one lane, it becomes possible to set an imaging range on each of the right side, the center and the left side of the lane, thereby enabling imaging. It is possible to capture a high-resolution (high-resolution) stereo image with a small number of road surface imaging cameras. Here, in particular, the road width is generally defined as 3.5 m. Accordingly, it is conceivable that both ends of the lane in the road width direction are imaged by the road surface imaging cameras 6L and 6R and the center portion is imaged by the road surface imaging camera 6C corresponding to the road width of 3.5 m.

  Note that the imaging system may be configured by using only one road surface imaging camera having an angle of view that allows the imaging range to cover the prescribed road width (3.5 m).

[Configuration of Imaging System According to First Embodiment]
Next, the configuration of the imaging system according to the first embodiment will be described. In the following description, it is assumed that the imaging system includes two road surface imaging cameras.

FIG. 8 is a block diagram illustrating an example of a schematic configuration of the imaging system according to the first embodiment. In FIG. 8, the imaging system 10 includes imaging units 100 ₁ and 100 ₂ , a position information acquisition unit 101, an imaging control unit 102, and a traveling direction imaging unit 103.

The imaging units 100 ₁ and 100 ₂ correspond to the above-described road surface imaging cameras 6L and 6R, respectively. The position information acquisition unit 101 acquires the current position of the vehicle 1. The traveling direction imaging unit 103 corresponds to the traveling direction camera 7 described above, images the road surface 4 ahead of the traveling direction of the vehicle 1, and acquires a traveling direction captured image.

The imaging control unit 102 controls imaging operations such as imaging timing, exposure, and shutter speed of the imaging units 100 ₁ and 100 ₂ , respectively. The imaging control unit 102 obtains the brightness of the area to be imaged by the imaging units 100 ₁ and 100 ₂ based on the traveling direction captured image acquired by the traveling direction imaging unit 103, and based on the obtained brightness, the imaging unit Imaging conditions including the exposure amount are set for 100 ₁ and 100 ₂ . Further, the imaging control unit 102 determines the timing for setting imaging conditions for the imaging units 100 ₁ and 100 ₂ based on the current position of the vehicle 1 acquired by the position information acquisition unit 101.

  FIG. 9 is a block diagram illustrating an example of a hardware configuration of the imaging system 10 according to the first embodiment. 9, the imaging system 10 includes road surface imaging cameras 6L and 6R (stereo cameras 6L and 6R), a traveling direction camera 7, an information processing device 50 corresponding to the PC 5 in FIG. 3, and a position information acquisition unit 51. And including.

  The information processing apparatus 50 generates a trigger at a predetermined timing, and sends the generated trigger to the road surface imaging cameras 6L and 6R. The road surface imaging cameras 6L and 6R perform imaging in synchronization with this trigger. The stereo captured images captured by the road surface imaging cameras 6L and 6R are supplied to the information processing apparatus 50. The information processing apparatus 50 stores and accumulates each stereo captured image supplied from the road surface imaging cameras 6L and 6R in a storage or the like. The information processing apparatus 50 performs image processing such as generation of a depth map and joining of the generated depth maps based on the accumulated stereo captured image.

  Further, the information processing apparatus 50 generates a trigger for instructing imaging of the traveling direction camera 7 and sends it to the traveling direction camera 7. This trigger does not need to be in tune with the trigger to be sent to the road surface imaging cameras 6L and 6R described above. The traveling direction camera 7 performs imaging in response to this trigger. The information processing apparatus 50 obtains an imaging condition including an exposure amount for the road surface imaging cameras 6L and 6R based on the traveling direction captured image supplied from the traveling direction camera 7, and uses the obtained imaging condition as setting information for road surface imaging. Supplied to the cameras 6L and 6R, respectively. Here, the information processing apparatus 50 can obtain imaging conditions for the road surface imaging camera 6L and the road surface imaging camera 6R, respectively. The information processing apparatus 50 supplies the setting information including the imaging conditions obtained for the road surface imaging cameras 6L and 6R to the road surface imaging cameras 6L and 6R, respectively. At this time, the information processing apparatus 50 determines the timing for setting the setting information for the road surface imaging cameras 6L and 6R based on the current position of the vehicle 1 acquired by the position information acquisition unit 51.

  FIG. 10 is a block diagram illustrating an exemplary configuration of a road surface imaging camera 6L according to the first embodiment. Note that the road surface imaging camera 6R can be realized with the same configuration as the road surface imaging camera 6L, and thus description thereof is omitted here.

10, the road surface imaging camera 6L includes an imaging optical system 600 _L and 600 _R, and the imaging device 601 _L and 601 _R, and a driving unit 602 _L and 602 _R, and a signal processing unit 603 _L and 603 _R, the output Part 604. Among these, the imaging optical system 600 _L , the imaging element 601 _L , the driving unit 602 _L , and the signal processing unit 603 _L have a configuration corresponding to the imaging lens 6L _L described above. Similarly, the imaging optical system 600 _R , the imaging element 601 _R , the driving unit 602 _R , and the signal processing unit 603 _R have a configuration corresponding to the imaging lens 6L _R described above.

The imaging optical system 600 _L is an optical system having an angle of view α and a focal length f, and projects light from the subject onto the imaging element 601 _L. The image sensor 601 _L is an optical sensor using, for example, a CMOS (Complementary Metal Oxide Semiconductor), and outputs a signal corresponding to the projected light. Note that an optical sensor using a CCD (Charge Coupled Device) may be applied to the image sensor 601 _L. Driver 602 _L drives the image sensor 601 _L, noise removal with respect to the signal output from the image sensor 601 _L, and outputs performs a predetermined processing such as gain adjustment. The signal processing unit 603 _L performs A / D conversion on the signal output from the driving unit 602 _L, and converts the signal into a digital image signal (captured image). The signal processing unit 603 _L performs predetermined image processing such as gamma correction on the converted image signal and outputs it. The captured image output from the signal processing unit 603 _L is supplied to the output unit 604.

The operations of the imaging optical system 600 _R , the imaging element 601 _R , the driving unit 602 _R , and the signal processing unit 603 _R are the same as the above-described imaging optical system 600 _L , imaging element 601 _L , driving unit 602 _L , and signal processing unit. Since this is the same as 603 _L , description thereof is omitted here.

For example, a trigger output from the information processing apparatus 50 is supplied to the drive units 602 _L and 602 _R. This trigger is supplied in synchronization with the drive units 602 _L and 602 _R included in the road surface imaging camera 6R. The driving units 602 _L and 602 _R take signals from the image sensors 601 _L and 601 _R according to this trigger and perform imaging.

Here, the driving units 602 _L and 602 _R perform the exposure in the image sensors 601 _L and 601 _R by the collective simultaneous exposure method. This method is called a global shutter. On the other hand, since the rolling shutter is a method of capturing light in order (line order) from the top of the pixel position, each line in the frame is not a photograph of a subject at exactly the same time. In the case of the rolling shutter system, if the camera or the subject moves at high speed while capturing an image signal of one frame, the image of the subject is captured with a shift depending on the line position. For this reason, in the road surface imaging cameras 6L and 6R according to the first embodiment, a global shutter is used so that the road shape is accurately imaged in terms of projection geometry.

Also, setting information output from the information processing apparatus 50 and including the imaging conditions relating to exposure is supplied to the drive units 602 _L and 602 _R and the signal processing units 603 _L and 603 _R , respectively. The drive units 602 _L and 602 _R set, for example, a gain and an exposure time according to this setting condition. In addition, the signal processing units 603 _L and 603 _R can set a gamma coefficient and a gradation at the time of gamma correction according to this setting condition.

The output unit 604 outputs the captured images of each frame supplied from the signal processing units 603 _L and 603 _R as a set of stereo captured images. The stereo captured image output from the output unit 604 is sent to the information processing apparatus 50 and accumulated.

  FIG. 11 is a block diagram illustrating a configuration example of the information processing apparatus 50 applicable to the first embodiment. 11, the information processing apparatus 50 includes a CPU (Central Processing Unit) 5000, a ROM (Read Only Memory) 5001, a RAM (Random Access Memory) 5002, and a graphics I / F (interface) connected to a bus 5030, respectively. ) 5003, storage 5004, input device 5005, data I / F 5006, and communication I / F 5007. Further, the information processing apparatus 50 includes camera I / Fs 5010a and 5010b, a speed acquisition unit 5021, and a position information acquisition unit 5022 that are connected to the bus 5030, respectively.

  The storage 5004 is a storage medium that stores data in a nonvolatile manner, and a hard disk drive or a flash memory can be applied. The storage 5004 stores programs and data for operating the CPU 5000.

  The CPU 5000 controls the overall operation of the information processing apparatus 50 using the RAM 5002 as a work memory, for example, according to a program stored in the ROM 5001 or the storage 5004 in advance. The graphics I / F 5003 generates a display signal that can be supported by the display 5020 based on the display control signal generated by the CPU 5000 according to the program. The display 5020 displays a screen corresponding to the display signal supplied from the graphics I / F 5003.

  The input device 5005 receives a user operation and outputs a control signal corresponding to the received user operation. As the input device 5005, a pointing device such as a mouse or a tablet, or a keyboard can be used. Alternatively, the input device 5005 and the display 5020 may be integrally formed to have a so-called touch panel configuration.

  The data I / F 5006 transmits / receives data to / from an external device. As the data I / F 5006, for example, USB (Universal Serial Bus) can be applied. A communication I / F 5007 controls communication with an external network in accordance with an instruction from the CPU 5000.

  The camera I / F 5010a is an interface to the road surface imaging cameras 6L and 6R. The stereo captured images output from the road surface imaging cameras 6L and 6R are transferred to, for example, the CPU 5000 via the camera I / F 5010a. The camera I / F 5010a generates the above-described trigger according to the instruction from the CPU 5000, and sends the generated trigger to the road surface imaging cameras 6L and 6R. Further, the camera I / F 5010a sends the setting information generated by the CPU 5000 to the road surface imaging cameras 6L and 6R.

  The camera I / F 5010b is an interface to the traveling direction camera 7. The travel direction captured image output from the travel direction camera 7 is transferred to, for example, the CPU 5000 via the camera I / F 5010b. In addition, the camera I / F 5010b sends a trigger, which is generated by the CPU 5000, for instructing the imaging timing to the traveling direction camera 7.

  The speed acquisition unit 5021 acquires speed information indicating the speed of the vehicle 1. When the road surface imaging cameras 6L and 6R are attached to the vehicle 1, the speed information acquired by the speed acquisition unit 5021 indicates the speed of the road surface imaging cameras 6L and 6R with respect to the subject (road surface). The speed acquisition unit 5021 has a function of receiving, for example, a GNSS (Global Navigation Satellite System) signal, and acquires speed information indicating the speed of the vehicle 1 based on the Doppler effect of the received GNSS signal. Not limited to this, the speed acquisition unit 5021 can also acquire speed information directly from the vehicle 1.

  The position information acquisition unit 5022 acquires position information indicating the current position of the vehicle 1. The position information acquisition unit 5022 acquires the position information using, for example, the reception function of the GNSS signal that the speed acquisition unit 5021 has.

  FIG. 12 is a functional block diagram of an example for explaining the functions of the information processing apparatus 50 applicable to the first embodiment. 12, the information processing apparatus 50 includes captured image acquisition units 500a and 500b, a UI unit 501, a control unit 502, a captured image storage unit 503, and an imaging control unit 504. The information processing apparatus 50 further includes a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a state characteristic value calculation unit 521, and a record creation unit 522.

  These captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, and record creation unit 522 include This is realized by a program operating on the CPU 5000. Without being limited thereto, the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, and record A part or all of the creation unit 522 may be configured by a hardware circuit that operates in cooperation with each other.

  The captured image acquisition unit 500a acquires stereo captured images from the road surface imaging cameras 6L and 6R. The captured image acquisition unit 500 b acquires a traveling direction captured image from the traveling direction camera 7. The captured image storage unit 503 stores the stereo captured image and the traveling direction captured image acquired by the captured image acquisition units 500a and 500b, for example, in the storage 5004. Also, the captured image storage unit 503 acquires the stored stereo captured image and traveling direction captured image from the storage 5004.

  The UI unit 501 realizes a user interface by display on the input device 5005 and the display 5020. The control unit 502 controls the overall operation of the information processing apparatus 50.

  The imaging control unit 504 corresponds to the position information acquisition unit 101 and the imaging control unit 102 described above. That is, the imaging control unit 504 generates an imaging condition of the traveling direction camera 7 based on the traveling direction captured image acquired by the captured image acquisition unit 500b. Further, the imaging control unit 504 generates a trigger for instructing the traveling direction camera 7 to perform imaging based on the position information indicating the current position of the vehicle 1 acquired by the position information acquisition unit 101.

  In addition, the imaging control unit 504 acquires speed information indicating the speed of the road surface imaging cameras 6L and 6R with respect to the subject (road surface 4). The acquired speed information and preset road surface imaging cameras 6L and 6R are acquired. Based on the angle of view α and the height h, a trigger for instructing synchronous imaging of the road surface imaging cameras 6L and 6R is generated.

  The matching processing unit 510 performs matching processing using the two captured images constituting the stereo captured image acquired by the captured image acquisition unit 500. The 3D information generation unit 511 performs processing related to 3D information. For example, the 3D information generation unit 511 obtains depth information by trigonometry or the like using the result of the matching processing by the matching processing unit 510, and generates 3D point cloud information based on the obtained depth information.

  The 3D information acquisition unit 520 acquires the 3D point cloud information obtained for each stereo captured image by the 3D information generation unit 511. The state characteristic value calculation unit 521 uses the three-dimensional point cloud information acquired by the 3D information acquisition unit 520 and the stereo captured images acquired by the captured image acquisition unit 500 to crack the MCI. Each state characteristic value of the rate C, the rutting amount D, and the flatness σ is calculated. The record creation unit 522 determines the MCI based on each state characteristic value calculated by the state characteristic value calculation unit 521 and creates a record.

  A program for realizing each function according to the first embodiment in the information processing apparatus 50 is a file in an installable format or an executable format, which is a CD (Compact Disk), a flexible disk (FD), a DVD (Digital Versatile). Disk) and the like are provided on a computer-readable recording medium. However, the present invention is not limited to this, and the program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. In addition, the program may be provided or distributed via a network such as the Internet.

  The program includes captured image acquisition units 500a and 500b, UI unit 501, control unit 502, captured image storage unit 503, imaging control unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristics. The module configuration includes a value calculation unit 521 and a record creation unit 522. As actual hardware, the CPU 5000 reads the program from a storage medium such as the storage 5004 and executes the program, whereby the above-described units are loaded on a main storage device such as the RAM 5002, and the captured image acquisition units 500a and 500b, UI Unit 501, control unit 502, captured image storage unit 503, imaging control unit 504, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, and record creation unit 522 It is supposed to be generated above.

[Trigger Generation Method Applicable to First Embodiment]
Next, a trigger generation method for instructing imaging to each of the road surface imaging cameras 6L and 6R that can be applied to the first embodiment will be described in more detail. In the first embodiment, the generation unit 103 indicates the distance in the speed direction of the vehicle 1 in the stereo imaging range of the road surface imaging cameras 6L and 6R with respect to the object to be measured (road surface 4). A trigger is generated at a time interval equal to or less than a predetermined time that is shorter than the time for moving at a speed.

  That is, the trigger needs to be generated so that the shooting range of the road surface 4 in the road surface imaging cameras 6L and 6R maintains a predetermined overlapping rate (traveling direction overlapping rate Dr) in the traveling direction of the vehicle 1. The purpose of this is to enable detection of sufficient corresponding points in order to stably calculate a highly accurate camera position in the process of calculating the imaging position from each stereo captured image, as will be described later. The lower limit value of the traveling direction overlap rate Dr is determined experimentally, for example, as “60%”.

In the first embodiment, the following three methods can be applied as a trigger generation method.
(1) Method of generating at regular time intervals (first generation method)
(2) Method of generating by detecting the moving speed of the camera (second generation method)
(3) A method of calculating and generating a movement distance using a captured image (third generation method)

(First generation method)
First, a first trigger generation method will be described. In the first generation method, the trigger time interval is determined from the maximum speed Speed of the vehicle 1 being imaged and the size of the imaging range (the length of the imaging range in the traveling direction of the vehicle 1). The speed acquisition unit 5021 acquires in advance the maximum speed Speed of the vehicle 1 based on a system setting value in the vehicle 1 or a user input to the information processing apparatus 50. The imaging control unit 504 acquires the maximum speed Speed from the speed acquisition unit 5021, and uses the following formula to determine the trigger time interval using the acquired maximum speed Speed, the traveling direction visual field Vp, and the traveling direction overlap rate Dr. Calculate by (4). The lower limit value described above is applied to the traveling direction overlap rate Dr.

  The number of triggers fps to be generated per second is calculated from the equation (4). The reciprocal of the trigger number fps is the time interval until the next trigger to be generated.

  As described with reference to FIG. 4, the traveling direction visual field Vp is typically the height h from the road surface 4 of each of the road surface imaging cameras 6L and 6R and the angle of view of each of the road surface imaging cameras 6L and 6R. It can be set based on α. Actually, the traveling direction visual field Vp is set in consideration of the angles of the road surface imaging cameras 6L and 6R with respect to the road surface 4 and the like.

  Here, when the moving vehicle 1 curves to the right or left, the movement amount of the camera outside the road surface imaging cameras 6L and 6R increases. Therefore, it is more preferable to use the rotational speed based on the position of the outer camera instead of using the maximum speed Speed of the vehicle 1 as it is.

  The imaging control unit 504 stores and accumulates all of the stereo captured images captured according to the generated trigger, for example, in the storage 5004 or the RAM 5002.

(Second generation method)
Next, the second trigger generation method will be described. While the first method described above is simple, when the vehicle 1 is stopped or moving at a speed lower than a predetermined speed, images are captured at an excessively fine interval with respect to the maximum speed Speed. As a result, the amount of the stereo captured image to be accumulated becomes large. In the second generation method, the imaging control unit 504 detects the moving speed of the camera, and generates a trigger according to the detected moving speed.

  The imaging control unit 504 calculates the time interval until the next trigger using the current speed of the vehicle 1 indicated by the speed information acquired by the speed acquisition unit 5021 as the maximum speed Speed of Expression (4), and performs imaging. I do. According to the second generation method, the lower the moving speed of the vehicle 1 is, the longer the trigger generation time interval is suppressed, and the useless imaging is suppressed.

  The imaging control unit 504 stores and accumulates all of the stereo captured images captured according to the generated trigger, for example, in the storage 5004 or the RAM 5002.

  The second generation method and the first generation method described above can be implemented in combination.

(Third generation method)
Next, a third trigger generation method will be described. In the third generation method, a trigger is generated at regular time intervals based on the maximum speed Speed of the vehicle 1 as in the first generation method described above. Here, in the third generation method, not all of the stereo captured images captured in response to the trigger are accumulated, but only when the traveling direction overlap rate Dr falls below a preset value.

  As will be described later, it is possible to calculate the moving distance of the camera (vehicle 1) using only stereo captured images that are captured so as to overlap in the traveling direction of the vehicle 1.

  The imaging control unit 504 calculates the moving distance of the camera using the last captured stereo captured image and the previously captured stereo captured image. The imaging control unit 504 determines whether or not the calculated distance exceeds the movement distance corresponding to the lower limit value of the traveling direction overlap rate Dr. When it is determined that the image capturing control unit 504 has exceeded, the image capturing control unit 504 accumulates the previously captured stereo image. On the other hand, if the imaging control unit 504 determines that it does not exceed, the stereo captured image captured immediately before is discarded.

  As a result, even if a sensor for measuring the current speed of the vehicle 1 is not used, useless image accumulation is not performed when the moving speed is low.

[Calculation method of road surface property value applicable to the first embodiment]
Next, a road surface property value calculation method applicable to the first embodiment will be described. Below, the measuring method of flatness (sigma), rutting amount D, and crack rate C applicable to 1st Embodiment is demonstrated.

(Flatness)
FIG. 13 is a flowchart illustrating an example of flatness calculation processing applicable to the first embodiment. In step S100, the captured image acquisition unit 500 acquires images captured by the road surface imaging cameras 6L and 6R, for example, stored in the storage 5004. If a stereo picked-up image is acquired, a process will transfer to step S101a and step S101b which can be processed in parallel.

  In step S101a, a depth map is generated based on the stereo captured image acquired in step S100. Depth map generation processing applicable to the first embodiment will be described with reference to FIGS. 14 and 15. FIG. 14 is a flowchart illustrating an example of a depth map generation process applicable to the first embodiment.

  In step S120, the matching processing unit 510 acquires a stereo captured image from the captured image acquisition unit 500. In the next step S121, the matching processing unit 510 performs a matching process based on the two captured images constituting the acquired stereo captured image. In the next step S122, the 3D information generation unit 511 calculates depth information based on the matching processing result in step S121, and generates a depth map that is three-dimensional point cloud information.

  The process of step S121 and step S122 will be described more specifically. In the first embodiment, depth information is calculated by a stereo method using two captured images constituting a stereo captured image. The stereo method here uses two captured images captured from different viewpoints by two cameras, and a pixel (reference pixel) in one captured image corresponds to a corresponding pixel ( This is a method of calculating a depth (depth value) by trigonometry based on a reference pixel and a corresponding pixel.

  In step S121, the matching processing unit 510 uses the two captured images constituting the stereo captured image acquired from the captured image acquisition unit 500, and has a predetermined size centered on the reference pixel in one of the reference captured images. A search is performed by moving a region in the other captured image corresponding to the region in the other captured image to be searched.

  Various methods are known for searching for corresponding pixels. For example, a block matching method can be applied. In the block matching method, a pixel value of an area cut out as a block of M pixels × N pixels around a reference pixel in one captured image is acquired. Further, in the other captured image, a pixel value of an area cut out as a block of M pixels × N pixels centering on the target pixel is acquired. Based on the pixel value, the similarity between the region including the reference pixel and the region including the target pixel is calculated. Similarities are compared while moving a block of M pixels × N pixels in the search target image, and the target pixel in the block with the highest similarity is set as a corresponding pixel corresponding to the reference pixel.

The similarity can be calculated by various calculation methods. For example, the normalized cross-correlation (NCC) shown in Equation (5) is one of cost functions, and the higher the value of the numerical value C _NCC indicating the cost, the higher the degree of similarity. Indicates. In equation (5), values M and N represent the size of a pixel block for performing a search. The value I (i, j) represents the pixel value of the pixel in the pixel block in one captured image serving as a reference, and the value T (i, j) represents the pixel block in the other captured image to be searched. Represents the pixel value.

As described above, the matching processing unit 510 moves the pixel block in the other captured image corresponding to the pixel block of M pixels × N pixels in one captured image, for example, in units of pixels in the other captured image. The calculation of Expression (5) is executed to calculate the numerical value C _NCC . In the other captured image, the center pixel of the pixel block having the maximum numerical value C _NCC is set as a target pixel corresponding to the reference pixel.

  Returning to the description of FIG. 14, in step S122, the 3D information generation unit 511 calculates a depth value (depth information) using trigonometry based on the reference pixel and the corresponding pixel obtained by the matching process in step S121. Then, three-dimensional point group information relating to one captured image and the other captured image that form the stereo captured image is generated.

FIG. 15 is a diagram for explaining trigonometry applicable to the first embodiment. The distance S to the target object 403 in FIG. (E.g. 1 point on the road 4), be calculated from the image pickup position information in the captured image on the imaging elements 402 (corresponding to the imaging device 601 _L and 601 _R) Is the purpose of processing. That is, this distance S corresponds to the depth information of the target pixel. The distance S is calculated by the following equation (6).

In formula (6), the value baseline represents the length of the base line (base line length) between the cameras 400a and 400b. In the example of FIG. 5, this corresponds to the baseline length by the imaging lenses 6L _L and 6L _R (in the case of the road surface imaging camera 6L). The value f represents the focal length of the lens 401 (corresponding to the imaging lenses 6L _L and 6L _R ). The value q represents the parallax. The parallax q is a value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel by the pixel pitch of the image sensor. The coordinate value of the corresponding pixel is obtained based on the result of the matching process in step S121.

This equation (6) is a calculation method of the distance S when the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R are used. This calculates the distance S from the captured images captured by the two cameras 400a and 400b, that is, the imaging lenses 6L _L and 6L _R , respectively. In the first embodiment, the calculation method of the equation (6), the imaging lens 6L _L and 6L _R road imaging camera 6L, and was imaged by the imaging lens 6R _L and 6R _R road imaging camera 6R Applying to each captured image, the distance S is calculated for each pixel.

  Returning to the description of FIG. 13, in step S101b, the 3D information generation unit 511 estimates the camera position and orientation. Here, the camera position indicates, for example, the center coordinates when the road surface imaging camera 6L is regarded as one body. Details of the processing in step S101b will be described later.

  When the processes of step S101a and step S101b are completed, the process proceeds to step S102. In step S102, the 3D information generation unit 511 calculates the previous depth map so that it matches the coordinate system of the previous time depth map based on the camera position and orientation estimated in step S101b for two depth maps adjacent in time series. The coordinate conversion of the depth map of the next time after the time is performed.

  Note that the previous time and the next time are the first imaging performed by the road imaging cameras 6L and 6R in the first imaging and the second imaging continuously executed in time series according to the trigger. The time at which the image was taken is the previous time, and the time when the second imaging is performed is the next time. That is, the depth map at the previous time is a depth map generated based on the stereo image captured by the first imaging, and the depth map at the next time is a depth map generated based on the stereo image captured by the second imaging. is there.

  In the next step S103, the 3D information generation unit 511 integrates the depth map of the next time coordinate-converted in step S102 into the depth map of the previous time. In other words, since the two depth maps of the previous time and the next time are arranged in a common coordinate system by the coordinate conversion in step S102, these two depth maps can be integrated. The processes of step S102 and step S103 are performed on the stereo captured images captured at all times.

  As shown in FIGS. 5 and 7, when a plurality of road surface imaging cameras 6L and 6R or road surface imaging cameras 6L, 6C and 6R are used side by side in the road width direction, the road surface images are aligned in the road width direction. The camera positions and orientations are similarly estimated between the cameras for use (for example, the road surface imaging cameras 6L and 6R), and then the depth maps are integrated.

  Further, the moving distance of the camera (vehicle 1) in the third generation method described above can be calculated in the coordinate transformation in step S102 and the depth map integration process in step S103.

  In the next step S104, the 3D information acquisition unit 520 acquires the depth map integrated in step S103 from the 3D information generation unit 511. Then, the state characteristic value calculation unit 521 calculates the flatness σ based on the depth map acquired by the 3D information acquisition unit 520. That is, when the depth maps are integrated, the road surface shape is restored as a point group in one three-dimensional space for the roads in the measured section. When the point cloud is restored, for each sampling point on the road surface, take the coordinate system connecting the coordinates of the 1.5m front and back points according to the regulations, and calculate the distance to the 3D point group in the center part, The displacement amount d can be calculated in the same manner as (2), and the flatness σ can be calculated from the displacement amount d according to the equation (3).

(Wad digging amount)
For the rutting amount D, in step S104, the state characteristic value calculation unit 521 scans the depth map acquired by the 3D information acquisition unit 520 in the road width direction, so that FIG. 1 (a) and FIG. Depths D ₁ and D ₂ as shown in FIG. Based on the acquired depths D ₁ and D ₂ and information on the cross section obtained by scanning the depth map, the rutting amount D can be obtained.

(Crack rate)
For example, the state characteristic value calculation unit 521 applies the stereo captured image acquired in step S100 to the depth map integrated in step S103. That is, the state characteristic value calculation unit 521 integrates the respective stereo captured images that are captured in the traveling direction of the vehicle 1 with the traveling direction overlap rate Dr. The state characteristic value calculation unit 521 sets a 50 cm mesh for the integrated image according to the rule, acquires information such as cracks and patching in each mesh by image analysis, and based on the acquired information, the crack rate C Is calculated.

(Camera position and orientation estimation process)
Next, the camera position and orientation estimation processing in step S101b in the flowchart of FIG. 13 described above will be described in more detail. FIG. 16 is a flowchart illustrating an example of camera position and orientation estimation processing that can be applied to the first embodiment. In the first embodiment, the camera position and orientation are estimated using SfM described above.

  In step S <b> 130, the 3D information generation unit 511 acquires a stereo captured image from the captured image acquisition unit 500. Here, the 3D information generation unit 511 acquires two stereo captured images that are continuously captured in time series (a stereo captured image at the previous time and a stereo captured image at the next time). Next, in step S131, the 3D information generation unit 511 extracts feature points from each acquired stereo captured image. This feature point extraction process is a process of finding a point that is easy to detect as a corresponding point of each stereo image, and typically, a point called a corner where there is a change in the image and the change is not uniform is detected. To do.

  In the next step S132, the 3D information generation unit 511 detects a point obtained by imaging the same location as the point extracted as the feature point in the stereo captured image at the previous time from the stereo captured image at the next time. For this detection process, a technique called optical flow or a technique called feature point matching represented by SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), or the like can be applied.

  In the next step S133, the 3D information generation unit 511 estimates the initial position and orientation of the camera. The 3D information generation unit 511 uses the coordinates of the corresponding points detected in each stereo captured image in step S132 as fixed values, and solves simultaneous equations using the position and orientation of the camera that captured the stereo captured image at the next time as parameters. Then, the position of the camera is estimated and three-dimensional coordinates are calculated (step S134).

A method for estimating the initial position and orientation of the camera will be described with reference to FIG. Expression (7) represents the relationship between coordinates x _{ij obtained} by projecting the point X _j of the space onto the camera (viewpoint) P _i . In Equation (7), the value n represents the number of points in the three-dimensional space, and the value m represents the number of cameras (captured images).

Each value P is a projection matrix for converting the coordinates of a point in the three-dimensional space into the two-dimensional coordinates of the image for each camera, and is represented by the following equation (8). Expression (8) is composed of a 2-by-2 conversion matrix of three-dimensional coordinates shown on the right side and a projective transformation f _i from the three-dimensional coordinates to the two-dimensional coordinates.

In FIG. 17, a value X _j and a value P _i are calculated using simultaneous equations given only coordinates x _ij reflected on the camera. A linear least square method can be used to solve the simultaneous equations.

  In the next step S135, the 3D information generation unit 511 optimizes the three-dimensional coordinates indicating the camera position calculated in step S134. The position and orientation of the camera calculated by solving the simultaneous equations may not have sufficient accuracy. Therefore, the accuracy is improved by performing the optimization process using the calculated value as the initial value.

Two-dimensional coordinates x _ij (reprojected) calculated by the above-described equation (7) using the corresponding point coordinates acquired by the search on the image, the three-dimensional coordinates calculated in step S134 and the projection matrix of the equation (8) as parameters. The difference between the two is called the residual. Parameters are adjusted by an optimization operation so that the sum of the residuals is minimized for all corresponding points in all stereo captured images used for calculating the position and orientation of the camera. This is called bundle adjustment. By performing overall optimization through bundle adjustment, the accuracy of the position and orientation of the camera can be improved.

The above is the base processing for camera position and orientation estimation in normal SfM. In the first embodiment, since the road surface imaging cameras 6L and 6R whose base line lengths are known are used, detection of corresponding points in a stereo captured image captured with overlapping imaging ranges according to the movement of the vehicle 1 is performed. In addition, it is easy to search for corresponding points in the two captured images constituting the stereo captured image. Accordingly, the three-dimensional coordinates X _j in the space are determined on the actual scale (actual size), and the position and orientation of the camera after movement can be calculated stably.

  As described above, in the imaging system 10 applicable to the first embodiment, the information processing apparatus 50 generates a trigger, and distributes the generated trigger to all the road surface imaging cameras 6L and 6R. Thus, the road surface imaging cameras 6L and 6R are configured to perform imaging in synchronization. At this time, a clock generator included in each of the road surface imaging cameras 6L and 6R, a trigger distribution component for distributing a trigger in the information processing device 50, and a trigger wiring length in the information processing device 50 and each of the road surface imaging cameras 6L and 6R There is a possibility that a slight deviation may occur in the timing at which the road surface imaging cameras 6L and 6R actually capture the trigger and execute the imaging due to the influence of the difference between the two. It is preferable to suppress this shift in imaging timing as much as possible.

  In particular, it is desirable to suppress the deviation of the imaging timing as much as possible between the road surface imaging cameras 6L and 6R whose base line length is known. For example, when wiring for trigger supply from the camera I / F 5010a to the road surface imaging cameras 6L and 6R, a relay is used to ensure the trigger quality, and a photocoupler is used to protect against electrostatic noise. There is a case. In this case, it is desirable to share these repeaters and photocouplers between the road surface imaging cameras 6L and 6R.

[Imaging Condition Determination Processing According to First Embodiment]
Next, the imaging condition determination process according to the first embodiment will be described in more detail. In the first embodiment, even when the stereo imaging range captured by the road surface imaging cameras 6L and 6R includes a large luminance change, the road surface imaging cameras 6L and 6R perform imaging under appropriate imaging conditions. It is possible.

  FIG. 18 is a diagram illustrating an example of a portion where there is a large luminance change on the road surface 4. FIG. 18A and FIG. 18B show examples of images taken while traveling from the bottom to the top of the figure.

  FIG. 18A is an example of a portion that enters the shade from the sun. The lower half of the image is a sunny area 41 and the upper half is a shaded area 40, and the shaded area 40 has lower road surface brightness than the sunny area 41. FIG. 18B is an example of a location where the road surface 4 is newly paved from the middle. The upper part of the image is a new pavement area 42 that has been newly paved at a place that has just been repaired. The new pavement area 42 has a lower reflectance than the other areas 43 of the road surface 4, and the road surface brightness is low.

  18A and 18B are both dark when an image is taken of a dark place (shade area 40, new pavement area 42) with the exposure setting at a bright place (sunlight area 41, other area 43). An image with sufficient contrast cannot be obtained at the location. Further, even when imaging is performed with automatic exposure, if imaging is performed while running on the vehicle 1, an image with insufficient contrast may be captured due to a time lag in tracking automatic exposure.

  Therefore, in the first embodiment, the stereo imaging range captured by the road surface imaging cameras 6L and 6R is imaged by the traveling direction camera 7 prior to the imaging by the road surface imaging cameras 6L and 6R, and obtained by imaging. Based on the obtained traveling direction captured image, the imaging conditions by the road surface imaging cameras 6L and 6R are determined.

  19 and 20 are flowcharts illustrating an example of the imaging condition determination process according to the first embodiment. Here, the advancing direction camera 7 and the road surface imaging cameras 6L and 6R are synchronized in imaging by a trigger supplied from the information processing apparatus 50. In the flowcharts of FIGS. 19 and 20, when one frame is imaged by the traveling direction camera 7, the imaging conditions of the road surface imaging cameras 6L and 6R are determined based on the traveling direction captured image obtained by the imaging. To do.

  First, processing according to the flowchart of FIG. 19 will be described. The flowchart in FIG. 19 illustrates processing related to the traveling direction camera 7 according to the first embodiment. In step S <b> 200, the information processing apparatus 50 acquires position information indicating the current position from the position information acquisition unit 51 by the imaging control unit 504. In the next step S <b> 201, the captured image acquisition unit 500 b acquires a traveling direction captured image captured by the traveling direction camera 7. The acquired traveling direction captured image is stored in the memory (for example, RAM 5002) of the information processing apparatus 50 by the captured image storage unit 503.

  In the next step S202, the information processing apparatus 50 causes the imaging control unit 504 to analyze the traveling direction captured image stored in step S201 and determine whether or not the imaging condition needs to be changed. The imaging control unit 504 calculates the luminance of the imaging range (imaging road surface) of the traveling direction camera 7 from the exposure setting of the traveling direction camera 7 and the image luminance in the attention area of the traveling direction captured image. Depending on the brightness, it is determined whether or not the imaging condition needs to be changed, and if so, what setting is to be set. Here, the imaging condition is, for example, exposure setting.

  For example, when the change in road surface luminance exceeds a threshold value, the imaging control unit 504 decreases the luminance so that the captured image becomes darker when the luminance changes in the direction of increasing brightness according to the amount of change. The exposure parameter is determined so that the captured image becomes bright. The exposure parameter is at least one of exposure time and gain.

  In step S203, when the imaging control unit 504 determines that the imaging condition needs to be changed by the determination process in step S202 (step S203, “Yes”), the imaging control unit 504 shifts the process to step S204. In step S204, the imaging control unit 504 associates the changed imaging condition with the changed position information indicating the position for changing the imaging condition, and stores the association in the RAM 5002, for example. This is because the shooting direction camera 7 and the road surface imaging cameras 6L and 6R have different imaging directions, so there is a time lag until the location imaged by the traveling direction camera 7 is captured by the road surface imaging camera. This is because it is necessary to change the exposure at an appropriate timing.

  Note that the changed position information is, for example, the position information acquisition unit 5021 (the information processing apparatus 50 or the GNSS receiver) of the imaging range captured by the traveling direction camera 7 in the position information acquired in step S200. A value obtained by adding the distance from can be applied.

  When the storage of the setting information and the changed position information in step S204 is completed, a series of processes according to the flowchart of FIG. If it is determined in step S203 described above that there is no need to change the exposure setting (step S203, “none”), the series of processes in the flowchart of FIG. 19 is terminated.

  The processes in steps S200 to S204 described above are repeatedly executed at predetermined time intervals.

  FIG. 20 is a flowchart illustrating an example of processing related to the road surface imaging cameras 6L and 6R according to the first embodiment. In addition, the information processing apparatus 50 acquires the position information frequently at a predetermined time interval by the position information acquisition unit 5022. In the information processing apparatus 50, the imaging control unit 504, based on the position information indicating the current position acquired by the position information acquisition unit 5022 in step S300, the changed position information stored in step S204 of FIG. It is determined whether or not the position is indicated by.

  If the imaging control unit 504 determines that the current position is the position indicated by the changed position information (step S300, “Yes”), the imaging control unit 504 shifts the process to step S301. In step S301, the imaging control unit 504 sets the setting information stored in association with the changed position information in step S204 of FIG. 19 for the road surface imaging cameras 6L and 6R. 6R imaging conditions are changed.

  In next step S302, the imaging control unit 504 generates a trigger for instructing imaging, sends the trigger to the road surface imaging cameras 6L and 6R, and each stereo imaged by the road surface imaging cameras 6L and 6R according to the trigger. A captured image is acquired. The imaging control unit 504 stores each acquired stereo captured image in, for example, the RAM 5002 and stores it in the storage 5004.

  On the other hand, if the imaging control unit 504 determines in step S300 that the current position is not the position indicated by the changed position information, the imaging control unit 504 shifts the process to step S302, generates a trigger for instructing imaging, and captures the road surface. The stereo captured images captured by the road surface imaging cameras 6L and 6R are acquired according to the trigger.

  The imaging system 10 according to the first embodiment repeatedly performs each process according to the flowcharts of FIGS. 19 and 20 while the vehicle 1 is moving, and acquires a stereo captured image of the road surface 4.

  According to the first embodiment, the luminance information of the stereo imaging range by the road surface imaging cameras 6L and 6R is acquired by the advancing direction camera 7 prior to the imaging of the road surface imaging cameras 6L and 6R, and the imaging conditions are set. Has been decided. Therefore, the imaging of the stereo imaging range by the road surface imaging cameras 6L and 6R can be executed under appropriate imaging conditions. As a result, the crack rate C, the rutting amount D, and the flatness σ can be calculated with higher accuracy based on the stereo images captured by the road surface imaging cameras 6L and 6R, and the reliability of the evaluation by MCI using these images It is possible to increase the sex.

[First Modification of First Embodiment]
Next, a first modification of the first embodiment will be described. The first modification of the first embodiment is provided in the imaging system 10 for each of a plurality of road surface imaging cameras (for example, road surface imaging cameras 6L and 6R) for imaging the road surface 4 for MCI calculation. Imaging conditions are set.

  The imaging condition determination process according to the first modification of the first embodiment will be described with reference to FIGS. Similarly to the above, the camera 7 for traveling direction and the camera cameras 6L and 6R for road surface imaging are synchronized in imaging by a trigger supplied from the information processing device 50. In the first modification of the first embodiment, when one frame is captured by the traveling direction camera 7, each of the road surface imaging cameras 6L and 6R is based on the traveling direction captured image obtained by the imaging. Determine imaging conditions.

  FIG. 21 is a flowchart illustrating an example of processing related to the traveling direction camera 7 according to the first modification of the first embodiment. In step S <b> 210, the information processing apparatus 50 acquires position information indicating the current position from the position information acquisition unit 51 by the imaging control unit 504. In the next step S <b> 211, the captured image acquisition unit 500 b acquires a traveling direction captured image captured by the traveling direction camera 7. The acquired traveling direction captured image is stored in the memory (for example, RAM 5002) of the information processing apparatus 50 by the captured image storage unit 503.

  In the next step S212, the information processing apparatus 50 causes the imaging control unit 504 to analyze the traveling direction captured image stored in step S211 and determine whether or not the imaging condition needs to be changed. The imaging control unit 504 calculates the luminance of the imaging range (imaging road surface) of the traveling direction camera 7 from the exposure setting of the traveling direction camera 7 and the image luminance in the attention area of the traveling direction captured image. Depending on the brightness, it is determined whether or not the imaging condition needs to be changed, and if so, what setting is to be set. Here, the imaging condition is, for example, an exposure parameter.

  For example, when the change in road surface luminance exceeds a threshold value, the imaging control unit 504 decreases the luminance so that the captured image becomes darker when the luminance changes in the direction of increasing brightness according to the amount of change. The exposure parameter is determined so that the captured image becomes bright. In the first modification of the first embodiment, the exposure parameters are individually determined for each of the road surface imaging cameras 6L and 6R.

  FIG. 22 is a diagram for explaining the relationship between the imaging range by the traveling direction camera 7 and the stereo imaging range by the road surface imaging cameras 6L and 6R, which can be applied to the first modification of the first embodiment. It is.

  As shown in FIG. 22, the road surface imaging cameras 6L and 6R are located behind the vehicle 1, the imaging surfaces 60L and 60R of the road surface imaging cameras 6L and 6R have overlapping portions, and the vehicle 1 travels. Imaging is performed so that the entire width direction of the lane on the inside road surface 4 falls within one of the imaging ranges 60L and 60R. For example, the road surface imaging cameras 6L and 6R are arranged such that either one of the imaging ranges 60L and 60R includes white lines 44 and 45 indicating both ends of the lane. Further, the traveling direction camera 7 performs imaging so that the entire width direction of the lane is within the imaging range 70 in front of the traveling direction of the vehicle 1.

  FIG. 23 is a diagram illustrating an example of images captured by the road surface imaging cameras 6L and 6R and the traveling direction camera 7 that can be applied to the first modification of the first embodiment. Note that, in FIG. 23, the same reference numerals are given to portions common to the above-described FIG. 23, and detailed description thereof is omitted. FIG. 23A corresponds to FIG. 23 described above, and FIG. 23B shows an example of a traveling direction captured image 71 obtained by capturing the imaging range 70 by the traveling direction camera 7. FIG. 23C illustrates an example of stereo captured images 61L and 61R obtained by the road surface imaging cameras 6L and 6R capturing the imaging ranges 60L and 60R.

  In determining the imaging conditions (exposure parameters) of the road surface imaging cameras 6L and 6R based on the traveling direction captured image of the traveling direction camera 7, an evaluation region for performing luminance evaluation on the traveling direction captured image 71 is: It is determined corresponding to each stereo image 61L and 61R. As shown in FIG. 23B, the evaluation areas 710L and 710R for performing luminance evaluation corresponding to the stereo captured images 61L and 61R are the centers of the imaging ranges 60L and 60R of the road surface imaging cameras 6L and 6R, respectively. Set according to the vicinity. The imaging conditions (exposure parameters) of the road surface imaging cameras 6L and 6R are individually determined according to the luminance information in the evaluation areas 710L and 710R.

  In step S213, when the imaging control unit 504 determines that the imaging condition needs to be changed by the determination process in step S212 (step S213, “Yes”), the process proceeds to step S214. In step S204, the imaging control unit 504 associates the changed imaging condition with the changed position information indicating the position for changing the imaging condition, for example, in the RAM 5002 for each of the road surface imaging cameras 6L and 6R. .

  When the storage of the setting information and the changed position information for each of the road surface imaging cameras 6L and 6R in step S214 is completed, a series of processes according to the flowchart of FIG. If it is determined in step S213 described above that there is no need to change the exposure setting (step S213, “none”), the series of processes in the flowchart of FIG. 21 is terminated.

  The processes in steps S210 to S214 described above are repeatedly executed at predetermined time intervals.

  FIG. 24 is a flowchart illustrating an example of processing related to the road surface imaging cameras 6L and 6R according to the first modification of the first embodiment. In the information processing apparatus 50, the imaging control unit 504, based on the position information indicating the current position acquired by the position information acquisition unit 5022 in step S310, the changed position information stored in step S214 of FIG. It is determined whether or not the position is indicated by.

  If the imaging control unit 504 determines that the current position is the position indicated by the changed position information (step S310, “Yes”), the imaging control unit 504 shifts the process to step S311. In step S301, the imaging control unit 504 stores each setting stored in association with the change position information for each of the road surface imaging cameras 6L and 6R in step S214 of FIG. 19 for the road surface imaging cameras 6L and 6R. Information is set, and the imaging conditions of the road surface imaging cameras 6L and 6R are changed.

  In the next step S312, the imaging control unit 504 generates a trigger for instructing imaging, sends it to the road surface imaging cameras 6L and 6R, and each stereo imaged by the road surface imaging cameras 6L and 6R according to the trigger. A captured image is acquired. The imaging control unit 504 stores each acquired stereo captured image in, for example, the RAM 5002 and stores it in the storage 5004.

  On the other hand, if the imaging control unit 504 determines in step S310 that the current position is not the position indicated by the changed position information, the imaging control unit 504 shifts the process to step S312 to generate a trigger for instructing imaging to generate road surface imaging. The stereo captured images captured by the road surface imaging cameras 6L and 6R are acquired according to the trigger.

  The imaging system 10 according to the first embodiment repeatedly performs each process according to the flowcharts of FIGS. 23 and 24 while the vehicle 1 is moving, and acquires a stereo captured image of the road surface 4.

  According to the first modification of the first embodiment, the moving direction camera 7 uses the traveling direction camera 7 to obtain the luminance information of the stereo imaging ranges of the road surface imaging cameras 6L and 6R prior to the imaging of the road surface imaging cameras 6L and 6R. And the imaging conditions are determined individually for the road surface imaging cameras 6L and 6R. Therefore, the imaging of the stereo imaging range by each of the road surface imaging cameras 6L and 6R can be executed under appropriate imaging conditions. As a result, the crack rate C, the rutting amount D, and the flatness σ can be calculated with higher accuracy based on the stereo images captured by the road surface imaging cameras 6L and 6R, and the reliability of the evaluation by MCI using these images It is possible to increase the sex.

[Second Modification of First Embodiment]
Next, a second modification of the first embodiment will be described. The second modification example of the first embodiment includes a foreign object detection process, and the process according to the first modification example of the first embodiment described above is more specific.

  FIG. 25 is a flowchart illustrating an example of processing related to the traveling direction camera 7 according to the second modification of the first embodiment. In step S <b> 400, the information processing apparatus 50 acquires position information indicating the current position from the position information acquisition unit 51 by the imaging control unit 504. In the next step S401, the captured image acquisition unit 500b acquires a traveling direction captured image captured by the traveling direction camera 7. The acquired traveling direction imaging is stored in the memory (for example, RAM 5002) of the information processing apparatus 50 by the captured image storage unit 503.

  In the next step S402, the imaging control unit 504 analyzes the traveling direction captured image stored in step S401, and detects an image indicating a foreign object from the traveling direction captured image. Here, the foreign matter refers to things other than road pavement such as white lines, road markings, manholes, fallen leaves, petals, earth and sand, snow, and dust. As a foreign matter detection method, for example, white line detection by straight line detection by Hough transform, and manhole (round shape) detection by circle detection by Hough transform can be applied. Further, as a foreign matter detection method, it is possible to detect other foreign matters based on the learning result of deep learning.

  In step S403, the imaging control unit 504 determines whether a foreign object is detected as a result of the process in step S402. If the imaging control unit 504 determines that a foreign object has been detected (step S403, “Yes”), the imaging control unit 504 shifts the process to step S404, and sets areas that do not include the foreign object image as the evaluation areas 710L and 710R. For example, when it is determined that the evaluation area 710L that is initially set corresponding to the stereo imaging range of the road surface imaging camera 6L includes a foreign object image, the imaging control unit 504 moves the position of the evaluation area 710L forward, backward, left, or right The evaluation area 710L does not include a foreign object image. Not limited to this, the evaluation region 710L may be reduced or deformed.

  On the other hand, if the imaging control unit 504 determines in step S403 that no foreign matter is detected (step S403, “None”), the process proceeds to step S405. In step S405, the imaging control unit 504 sets the original positions of the evaluation regions 710L and 710R in the traveling direction captured image 71, for example, the central region of the traveling direction captured image 71, in the evaluation regions 710L and 710R.

  After the process of step S404 or step S405, the process proceeds to step S410. In step S410, the imaging control unit 504 obtains the brightness of the evaluation areas 710L and 710R based on the pixels included in the evaluation areas 710L and 710R determined in step S404 or S405, and the obtained brightness is an appropriate value determined in advance. It is determined whether or not the brightness. When the imaging processing unit 504 determines that the luminance is appropriate (step S410, “Yes”), the series of processing according to the flowchart of FIG. 25 is terminated.

  If the imaging control unit 504 determines in step S410 that the luminance is not appropriate, the imaging control unit 504 shifts the process to step S411. In step S411, the imaging control unit 504 determines whether the luminance values of the evaluation areas 710L and 710R are brighter than a predetermined value. If the imaging control unit 504 determines that the luminance indicates a value brighter than a predetermined value, the imaging control unit 504 shifts the processing to step S412. In step S412, the imaging control unit 504 changes the imaging condition (exposure parameter) in a direction to darken the captured image.

  On the other hand, if the imaging control unit 504 determines in step S411 that the luminance is darker than the predetermined value, the imaging control unit 504 shifts the processing to step S413. In step S413, the imaging control unit 504 changes the imaging condition (exposure parameter) in a direction to brighten the captured image.

  The imaging control unit 504 shifts the process to step S414 after the process of step S412 or step S413. In step S414, the imaging control unit 504 associates the imaging condition changed in step S412 or step S413 with the changed position information indicating the position for changing the imaging condition, and stores the association in the RAM 5002, for example. When the storage of the setting information and the changed position information in step S414 is completed, a series of processes according to the flowchart of FIG.

  According to the second modified example of the first embodiment, the traveling direction camera 7 uses the road surface imaging cameras 6L and 6R to obtain the luminance information of the stereo imaging range prior to the imaging of the road surface imaging cameras 6L and 6R. Obtaining and determining imaging conditions. At that time, when the evaluation area 710L and 710R in the traveling direction captured image 71 captured by the traveling direction camera 7 includes a foreign object image, the imaging control unit 504 avoids the foreign object image and evaluates the evaluation area 710L. And 710R are set. Therefore, the imaging of the stereo imaging range by the road surface imaging cameras 6L and 6R can be executed under appropriate imaging conditions without being affected by the image of the foreign matter. As a result, the crack rate C, the rutting amount D, and the flatness σ can be calculated with higher accuracy based on the stereo images captured by the road surface imaging cameras 6L and 6R, and the reliability of the evaluation by MCI using these images It is possible to increase the sex.

[Second Embodiment]
Next, a second embodiment will be described. The second embodiment is an example in which an optical sensor that detects light of an object is used in place of the above-described moving direction camera 7. As the optical sensor, a photodiode or a phototransistor that simply detects light from an object may be used, or a laser scanner or the like may be used.

  FIG. 26 is a flowchart illustrating an example of imaging condition setting processing according to the second embodiment. In the flowchart of FIG. 26, the same reference numerals are given to the processes common to the flowchart of FIG. 19 described above, and detailed description thereof is omitted. Note that the output of the optical sensor is acquired by the captured image acquisition unit 500b as described above. In step S <b> 200, the information processing apparatus 50 acquires position information indicating the current position from the position information acquisition unit 51 by the imaging control unit 504. In the next step S221, the captured image acquisition unit 500b acquires the output of the optical sensor. The acquired output value is stored in the memory (for example, RAM 5002) of the information processing apparatus 50 by the captured image storage unit 503.

  In the next step S222, the imaging control unit 504 analyzes the output value of the optical sensor stored in step S221, and determines whether or not the imaging condition needs to be changed. The imaging control unit 504 calculates the luminance of the detection range (imaging road surface) based on the output value of the optical sensor, and whether or not the imaging condition (exposure parameter) needs to be changed according to the calculated luminance. If so, decide what settings to use.

  For example, when the change in road surface luminance exceeds a threshold value, the imaging control unit 504 decreases the luminance so that the captured image becomes darker when the luminance changes in the direction of increasing brightness according to the amount of change. The exposure parameter is determined so that the captured image becomes bright. The exposure parameter is at least one of exposure time and gain.

  In step S203, when the imaging control unit 504 determines that the imaging condition needs to be changed by the determination process in step S222 (step S203, “Yes”), the imaging control unit 504 shifts the process to step S204. In step S204, the imaging control unit 504 associates the changed imaging condition with the changed position information indicating the position for changing the imaging condition, and stores the association in the RAM 5002, for example. This is because the direction of the detection range of the optical sensor is different from the direction of the stereo imaging range of the road surface imaging cameras 6L and 6R. Therefore, the location where the luminance is detected by the optical sensor is captured by the road surface imaging cameras 6L and 6R. This is because there is a time lag and it is necessary to change the exposure at an appropriate timing.

  As described above, the changed position information is, for example, a value obtained by adding the distance from the position information acquisition unit 5021 in the detection range that is a detection target of the optical sensor to the position information acquired in step S200. Can be applied.

  When the storage of the setting information and the changed position information in step S504 is completed, a series of processes according to the flowchart of FIG. If it is determined in step S203 described above that there is no need to change the exposure setting (step S203, “none”), the series of processes in the flowchart of FIG. 26 is terminated.

  The processes of step S200, step S221, step S222, step S203, and step S204 described above are repeatedly executed at predetermined time intervals.

  Note that the processing related to the road surface imaging cameras 6L and 6R is the same as the processing described using, for example, FIG.

  According to the second embodiment, in order to detect the luminance of the evaluation area ahead of the vehicle 1 in the traveling direction, an optical sensor that can be realized with a simpler configuration than the traveling direction camera 7 is used. As a result, the imaging system can be configured on a smaller scale.

  Each of the above-described embodiments is a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the gist of the present invention.

1 Vehicle 4 Road surface 5 PC
6, 6C, 6L, 6R Stereo camera 6L _L , 6L _R , 6R _L , 6R _R Imaging lens 7 Traveling direction camera 10 Imaging system 50 Information processing device 51, 5022 Position information acquisition unit 60C, 60L, 60R Stereo imaging range 60C _{L 1} , 60 _{C R} , 60 _{L L} , 60 L _R , 60 R _L , 60 _{R R} Imaging range 100 ₁ , 100 ₂ Imaging unit 102, 504 Imaging control unit 103 Generation unit 500 a, 500 b Captured image acquisition unit 503 Captured image storage unit 710 L, 710 R Evaluation Region 5021 Speed acquisition unit

JP 2013-096740 A

Claims (10)

A first imaging device mounted on a moving body and imaging a road surface on which the moving body moves;
A second imaging device mounted on the movable body and imaging the road surface on the traveling direction side of the movable body with respect to the first imaging device;
An imaging control unit that determines imaging conditions of the first imaging device based on a captured image captured by the second imaging device;
An imaging system comprising: The imaging control unit
The imaging system according to claim 1, further determining timing for setting the imaging condition in the first imaging device. An acquisition unit for acquiring current position information indicating a current position of the mobile body;
The imaging control unit
The imaging system according to claim 2, wherein the timing is determined based on the current position information and a predetermined distance from the moving body of an imaging range by the second imaging device. A plurality of the first imaging devices are mounted on the moving body,
The imaging control unit
The imaging condition of each of the plurality of first imaging devices is determined based on an image of each region corresponding to the imaging range of each of the plurality of first imaging devices in the imaging range of the second imaging device. The imaging system according to any one of claims 1 to 3. The imaging control unit
2. The imaging condition is determined using an image of a region of the second captured image excluding the image of the foreign object when the captured image includes an image of the foreign object with respect to the road surface. 5. The imaging system according to any one of 4 above. An imaging method for an imaging system including a first imaging device and a second imaging device each mounted on a moving body,
The first imaging device images a road surface on which the moving body moves,
The second imaging device images the road surface on the traveling direction side of the moving body with respect to the first imaging device,
An imaging method including an imaging control step of determining an imaging condition of the first imaging device based on a captured image captured by the second imaging device.   A moving body on which the imaging system according to any one of claims 1 to 5 is mounted. An imaging unit for imaging the first imaging range;
An imaging control unit that determines an imaging condition of the imaging unit based on a captured image obtained by imaging a second imaging range different from the first imaging range;
An imaging apparatus comprising: The imaging apparatus according to claim 8, further comprising an acquisition unit that acquires the captured image obtained by capturing an image of the second imaging range by another imaging unit different from the imaging unit.   A moving body on which the imaging device according to claim 8 is mounted.