JP6551623B1 - Information processing apparatus, moving body, image processing system, and information processing method - Google Patents

Abstract

It is possible to measure each or a combination of three indicators in a running surface inspection with a simple configuration. An information processing apparatus according to an embodiment obtains a plurality of captured images including distance information in a depth direction in which a traveling surface of a moving object is imaged by a plurality of stereo imaging means; Image processing means for overlapping the captured images and connecting them in the width direction of the running surface. [Selection] Figure 9

Description

The present invention relates to an information processing apparatus, moving body, an image processing system and an information processing method.

  Since paved roads are damaged by vehicle traffic and weather, it is necessary to periodically inspect road (traveling surface) properties. Conventionally, various methods have been proposed for the inspection of the road property. Three indicators of the number of cracks, the depth of rutting, and flatness are defined as inspection items of the road property. For these indexes, it is known to visually observe or analyze a camera image for cracks, and to measure rutting by a light cutting method using a camera and a line scan laser. In addition, with regard to flatness, measurement using an instrument having a length of 1.5 m and a length of 3 m in total, which is called a profile meter, has been generally performed.

  Since the profile meter is manually moved, the measurement preparation is complicated, and there is a problem that quick measurement and movement are difficult. Patent Document 1 discloses a technology that allows a device for measuring flatness to be configured in a special vehicle having a length of 3 m or more in the traveling direction. According to Patent Document 1, a device for irradiating a laser below is installed at the front, middle, and rear of a special vehicle, and by measuring the distance to the ground at three locations simultaneously, the flatness can be measured efficiently. It is possible to

  However, the conventional road property inspection requires a dedicated device for each of the measurement targets related to the three indicators such as flatness measurement, rutting measurement, and crack analysis. Therefore, a special vehicle having a traveling direction length of 3 m or more described in Patent Document 1, a laser irradiation and measurement device for a light cutting method, and an illumination for image photographing are required, which are very expensive. There was a problem of becoming a system.

  The present invention has been made in view of the above, and an object of the present invention is to enable measurement of each or a combination of three indicators in a running surface inspection with a simple configuration.

In order to solve the problems described above and to achieve the object, the present invention provides a predetermined imaging range as a width direction of a traveling surface of a moving object by a plurality of stereo imaging units configured by imaging elements including pixels of a plurality of lines . Acquisition means for acquiring a plurality of captured images including distance information in the depth direction captured at an overlapping rate, a plurality of depth maps are generated for the plurality of captured images acquired, and the relative sizes of the plurality of depth maps are generated. Image processing means for performing a matching process to match the physical position, connecting a plurality of depth maps aligned by the matching process in the direction of the width of the traveling surface, and combining them into one depth map; comprising measuring means for measuring the amount, and the matching process performed by the image processing means the overlapping of the plurality of depth map area Said plurality of depth maps synthesized single depth map in accordance with the position in which both the luminance of the depth distance and pixel corresponds, said measuring means, width direction of the running surface resulting from one depth map obtained by the synthetic Based on the distance information, the rubbing amount of the running surface is measured.

  According to the present invention, there is an effect that it is possible to measure each or a combination of three indicators in the traveling surface inspection with a simple configuration.

FIG. 1 is a view showing an example of a mesh image for crack determination. FIG. 2 is a diagram for explaining the measurement of rutting amount. FIG. 3 is a diagram for explaining the measurement of flatness. FIG. 4 is a view showing an example of the arrangement of an image processing system according to the first embodiment. FIG. 5 is a diagram for explaining an imaging range in the traveling direction of the vehicle by the stereo camera, which can be applied to the first embodiment. FIG. 6 is a view for explaining an imaging range in the road width direction of a vehicle by a stereo camera, which is applicable to the first embodiment. FIG. 7 is a diagram showing an overlap in the traveling direction of the vehicle of the stereo imaging range by the stereo camera, which is applicable to the first embodiment. FIG. 8 is a diagram illustrating an example of an image processing system including three stereo cameras according to the first embodiment. FIG. 9 is a block diagram illustrating an example of a schematic configuration of the image processing system according to the first embodiment. FIG. 10 is a functional block diagram of an example for explaining functions of the imaging apparatus according to the first embodiment. FIG. 11 is a block diagram illustrating an example of a hardware configuration of the image processing system according to the first embodiment. FIG. 12 is a block diagram illustrating a configuration of an example of the stereo camera according to the first embodiment. FIG. 13 is a block diagram illustrating a configuration example of an information processing apparatus applicable to the first embodiment. FIG. 14 is a functional block diagram illustrating an example of functions of the information processing apparatus applicable to the first embodiment. FIG. 15 is a flowchart illustrating an example of flatness calculation processing according to the first embodiment. FIG. 16 is a flowchart illustrating an example of a depth map generation process applicable to the first embodiment. FIG. 17 is a diagram for explaining trigonometry applicable to the first embodiment. FIG. 18 is a diagram illustrating an example of a road surface image. FIG. 19 is a diagram showing an example of measurement of rut value. FIG. 20 is a flowchart illustrating an example of camera position and orientation estimation processing applicable to the first embodiment. FIG. 21 is a diagram for describing camera position and orientation estimation processing applicable to the first embodiment. FIG. 22 is a diagram illustrating another configuration example of the image processing system according to the first embodiment. FIG. 23 is a block diagram illustrating an example of a hardware configuration of an image processing system according to the second embodiment. FIG. 24 is a block diagram illustrating a configuration of an example of a stereo camera according to the second embodiment. FIG. 25 is a block diagram illustrating a configuration example of an information processing apparatus applicable to the second embodiment. FIG. 26 is a flowchart illustrating a flow of information processing in the image processing unit. FIG. 27 is a flowchart showing the flow of the connection process of step S2.

With reference to the accompanying drawings, an information processing apparatus, moving body, an embodiment of an image processing system and information processing method in detail.

[Outline of existing road condition inspection method]
Prior to the description of the embodiment, an existing road property inspection method will be schematically described. The Maintenance Control Index (MCI) of the pavement is defined as an index for evaluating road (traveling surface) properties. MCI quantitatively evaluates the serviceability of pavement based on three types of road surface property values: “cracking rate”, “rull amount” and “flatness”. Since roads (traveling surfaces) are inspected, roads (traveling surfaces) are also defined as inspection surfaces, inspection target surfaces, and the like. In the present specification, a road is also defined as a paved surface, and the paved surface means, for example, a paved road surface on which a passenger car travels.

  FIG. 1 is a view showing an example of a mesh image for crack determination. The example illustrated in FIG. 1 is an example in which the captured image of the road surface (traveling surface) 4 near the pedestrian crossing is divided into meshes of 50 cm. As shown in FIG. 1, cracks and the like are generated on the road surface near the pedestrian crossing.

  Of the three types of road surface property values, the “crack rate” is calculated as follows for each area where the road surface is divided into 50 cm meshes, according to the number of cracks and patching area. Apply to equation (1) to determine the crack rate. In patching, pot holes (also called pit holes or graze holes, pits or pits that are formed by peeling or sinking the surface layer) or cracks that are generated on the road surface are filled with repair material such as asphalt mixture or piled up. This refers to the location or method of repair that partially repaired the road surface.

The “cracking rate” can be calculated as follows based on inspection standards determined by administrative agencies and the like.
Two crack one ⇒0.15M ² Hibihibiware ⇒0.25M ² Nohibi patching area 0 to 25% ⇒ cracks 0 m ²
Patching area 25 to 75% ひ び 割 れ crack 0.125m ²
Patching area 75% or more ⇒ Crack 0.25m ²

  As shown in FIG. 2A and FIG. 2B, the “wadder digging amount” measures the depths D1 and D2 of two “wadachi” generated in one lane, and the measured depth The larger value is adopted among D1 and D2. There are several patterns for digging "Wadachi", so a measurement method is defined for each pattern. FIG. 2A shows a case where the distance between two “wadachi” is higher than both ends of the two “wadachi”, and FIG. The example of the measurement method in the case where it is lower than the both ends of "ruth" is shown.

  “Flatness” measures three heights from the reference plane at 1.5 m intervals along the traveling direction of the vehicle on the road surface. For example, as shown in FIG. 3, three points A, B, and C and measuring instruments d1 to d3 are provided on the lower surface of the vehicle at intervals of 1.5 m, and the heights X1, X2, and X3 are measured. Based on the measured heights X1, X2 and X3, the displacement amount d is obtained by the equation (2). This measurement is performed a plurality of times while moving the vehicle. Formula (3) is calculated based on the plurality of displacement amounts d obtained in this way, and flatness σ is calculated.

  The above-described measurement of “cracking rate”, “wad digging amount” and “flatness” is executed for each evaluation section of 100 m. When the damaged part is known in advance, the measurement may be performed by dividing 100 m into smaller units such as 40 m + 60 m. The MCI is calculated based on the measurement result executed in units of 100 m, and a record is created.

  MCI is calculated based on the measured crack rate C [%], rutting amount D [mm], and flatness σ [mm], and the four equations shown in Table 1 are calculated to obtain the values MCI, MCI1, MCI2 and Calculate MCI3. Then, the minimum value among the calculated values MCI, MCI1, MCI2 and MCI3 is adopted as MCI. Evaluation is performed according to the evaluation criteria shown in Table 2 based on the adopted MCI, and it is determined whether or not the road surface in the evaluation section needs repair.

First Embodiment
Next, the image processing system according to the first embodiment will be described. In 1st Embodiment, the camera (stereo camera) in which a stereo imaging is possible is attached to a vehicle, and a road surface is imaged. Based on the captured stereo image, depth information (depth distance) with respect to the road surface is acquired from the imaging position to generate a three-dimensional shape of the road surface, and three-dimensional road surface data is created. By analyzing this three-dimensional road surface data, it is possible to obtain “cracking ratio”, “wedge rubbing amount” and “flatness” used for obtaining MCI. A stereo camera is also called an imaging unit or a measuring device.

  It will be described more specifically. The stereo camera includes two cameras provided with a predetermined length (referred to as a baseline length) apart from each other, and outputs two pairs of captured images (referred to as stereo captured images) captured by the two cameras. . By searching for a corresponding point between two captured images included in this stereo captured image, the depth distance can be restored for any point in the captured image. Data in which the depth distance is restored for the entire area of the captured image and each pixel is represented by the depth distance is referred to as a depth map. That is, the depth map is three-dimensional point group information including a set of points each having three-dimensional information.

  Note that the depth distance refers to the height (distance) from the stereo camera to the road surface (travel surface). More specifically, it is the height (distance) from the imaging surface (imaging device surface) of the stereo camera to the road surface. Note that where the depth distance from the road surface, not the imaging surface of the stereo camera, can be appropriately changed by calculation.

  The depth direction is illustrated in FIG. The depth direction is generally a vertical direction with respect to the traveling surface (a direction perpendicular to the horizontal plane), but is not necessarily vertical. It can also be defined as the depth direction of a stereo image taken of the traveling surface, that is, the direction along the optical axis of the stereo camera. The depth direction includes at least a direction in which the rutting amount can be measured. Specifically, the direction of the rutting depths D1 and D2 shown in FIG. The distance information in the depth direction refers to height (distance) information from the stereo camera to the road (traveling surface) in a direction in which at least the rutting amount can be measured along the optical axis of the stereo camera.

  This stereo camera is attached to one place such as the rear of the vehicle so that the ground can be imaged, and the vehicle is moved along the road to be measured. For the purpose of explanation, in the stereo camera mounted on a vehicle for measurement, it is assumed that the imaging range covers a prescribed length in the road width direction.

  The rutting amount D arranges the depth distances of the portions cut out in a stripe shape in the road width direction in the depth map restored from the stereo captured image captured by the stereo camera. Based on the change of the depth distance in the road width direction, the rutting depths D1 and D2 can be calculated.

  The crack rate C is obtained by analyzing a picked-up image obtained by picking up an image of the road surface, detecting “cracks” and the like, and performing the calculation according to the above formula (1) based on the detection result.

  Here, in the traveling direction of the vehicle, since the imaging range by one imaging is limited, for example, the crack rate C in the 100 m section cannot be calculated based on the imaging image by one imaging. Therefore, imaging is sequentially performed every movement according to the length of the vehicle in the imaging range in the imaging range while moving the vehicle along the road. At this time, the imaging trigger (imaging trigger) is controlled so that the imaging range in the previous imaging and the imaging range in the current imaging overlap at an overlap rate designed in advance or higher.

  In this way, by controlling the imaging trigger according to the movement of the vehicle, the road surface of the road to be measured can be imaged without omission. Therefore, the picked-up images sequentially picked up according to the progress of the vehicle are connected using image processing called stitching to generate one image including, for example, a road surface image of a 100 m section. By visually confirming or analyzing this image, it is possible to determine the presence or absence of a crack on the road surface and to measure the crack rate C on the road surface.

  The flatness measurement uses a technique called Structure from Motion (hereinafter referred to as SfM) that estimates the imaging position based on images that are sufficiently overlapped from images at different imaging points.

  An overview of SfM processing will be described. First, using an image captured with overlapping imaging ranges, a point at which the same point is imaged in each image is detected as a corresponding point. It is desirable to detect as many corresponding points as possible. Next, for example, a simultaneous equation using the coordinates of the detected corresponding points is set for the movement of the camera from the imaging point of the first image to the imaging point of the second image, and the total error becomes the smallest. Find such parameters. Thus, the imaging position of the second image can be calculated.

  As described above, the depth distance of an arbitrary point in the image can be restored as a depth map from the stereo captured image at each imaging point. A depth map is a depth map that represents a depth distance between a shooting position of a stereo camera and a shooting target in a stereo image. For the depth map corresponding to the first stereo image, the depth map corresponding to the second stereo image is the depth with the imaging position of the second camera obtained from the above simultaneous equations as the origin. Coordinate conversion to map. Thus, the two depth maps can be unified with the coordinate system of the first depth map. In other words, a single depth map can be generated by combining two depth maps.

  By performing this processing on, for example, depth maps based on all stereo images captured in the 100 m section, the road surface of the 100 m section is restored in one three-dimensional space. The displacement amount d can be calculated by applying the depth distance of the road surface restored in this way to the above equation (2). The flatness σ is calculated by applying the displacement amount d to the above equation (3). The depth distance is a height X measured from the measuring instrument to the road surface (traveling surface).

  In the first embodiment, road surface flatness σ, rutting amount D for obtaining MCI by performing imaging with a stereo camera mounted on a vehicle and performing image processing on the captured stereo image, and The crack rate C can be measured collectively. In the first embodiment, this measurement can be performed using an image processing system including a stereo camera and an information processing apparatus that performs image processing on a stereo captured image, and the MCI can be obtained with a simple configuration. Become.

  In addition, in the existing technology, the measurement related to the three indexes for obtaining the MCI is executed by separate devices, so that control and management such as data storage and time synchronization between the data are complicated. . On the other hand, in the first embodiment, since three indexes for obtaining MCI are calculated based on a common stereo image, control and management such as data storage and time synchronization between data are easy. Become.

[Camera Arrangement Applicable to First Embodiment]
Next, an example of a camera arrangement applicable to the first embodiment will be described. FIG. 4 is a diagram illustrating a configuration example of the image processing system according to the first embodiment. FIG. 4A is a diagram illustrating a state in which the image processing system according to the first embodiment is mounted on the vehicle 1 from the side surface of the vehicle 1. In FIG. 4A, the direction toward the left end of the figure is the traveling direction of the vehicle 1. That is, in FIG. 4A, the left end side of the vehicle 1 is the front portion of the vehicle 1, and the right end side is the rear portion of the vehicle 1. FIG. 4B is a view showing an example of the vehicle 1 as viewed from the rear side.

  The image processing system according to the first embodiment includes at least a plurality of stereo cameras 6 as stereo imaging means and a personal computer (PC) 5 as an information processing apparatus. Furthermore, the image processing system may include the vehicle 1 as a moving body. The vehicle 1 (moving body, movable machine) is fixed so that the stereo camera 6 and the housing 1a that is the vehicle body can be fixed so that the stereo camera 6 can image the road (traveling surface) with respect to the housing 1a. The moving unit 1 b includes an engine, a tire, and the like that can move the housing 2 a along the road (traveling surface). The image processing system further includes a mounting member 3 provided with a fixing portion 2 at the rear of the vehicle body (housing 1 a) of the vehicle 1. The mounting member 3 mounts one or more stereo cameras 6 to the fixed portion 2. Here, as illustrated in FIG. 4B, two stereo cameras 6 </ b> L and 6 </ b> R functioning as a plurality of stereo imaging means are attached to both ends in the width direction of the vehicle body of the vehicle 1. Each of the stereo cameras 6L and 6R is attached in a direction for imaging the road surface 4 on which the vehicle 1 moves. Preferably, each stereo camera 6L and 6R is attached so as to image the road surface 4 from the vertical direction.

  Hereinafter, when it is not necessary to distinguish between the stereo cameras 6L and 6R, the stereo cameras 6L and 6R are collectively described as the stereo camera 6.

  The stereo camera 6 is controlled by, for example, the PC 5 installed inside the vehicle 1, for example. The operator operates the PC 5 to instruct start of imaging by the stereo camera 6. When the start of imaging is instructed, the PC 5 starts imaging with the stereo camera 6. The imaging is repeatedly executed with timing controlled according to the moving speed of the stereo camera 6, that is, the vehicle 1.

  For example, the operator operates the PC 5 in response to the end of imaging in a necessary section and instructs the end of imaging. The PC 5 ends the imaging by the stereo camera 6 in response to the imaging end instruction.

  FIG. 5 and FIG. 6 are diagrams showing examples of the imaging range of the stereo camera 6 applicable to the first embodiment. 5 and 6, the same reference numerals are given to the portions common to FIG. 4 described above, and the detailed description thereof is omitted.

  FIG. 5 is a view for explaining an imaging range (referred to as a traveling direction view Vp) of the traveling direction of the vehicle 1 by the stereo camera 6 applicable to the first embodiment. As in FIG. 2 described above, in FIG. 5, the direction toward the left end of the figure is the traveling direction of the vehicle 1. The traveling direction view Vp is determined according to the angle of view α of the stereo camera 6 and the height h of the stereo camera 6 with respect to the road surface 4 as shown in FIG.

  FIG. 6 is a diagram for explaining an imaging range in the road width direction of the vehicle 1 by the stereo camera 6 applicable to the first embodiment. FIG. 6 is a view of the vehicle 1 as viewed from the rear side. The same reference numerals as in FIG. 4 (b) denote the same parts as in FIG. 4 (b), and a detailed description will be omitted.

  In FIG. 6A, the stereo camera 6L includes two imaging lenses 6LL and 6LR. A line connecting the imaging lenses 6LL and 6LR is referred to as a base line, and the length thereof is referred to as a base line length. The stereo camera 6L is arranged so that the base line is perpendicular to the traveling direction of the vehicle 1. Similarly, the stereo camera 6R is also provided with two imaging lenses 6RL and 6RR separated by a baseline length, and the baseline is disposed so as to be perpendicular to the traveling direction of the vehicle 1.

  FIG. 6B shows an example of the imaging range of the stereo cameras 6L and 6R applicable to the first embodiment. In the stereo camera 6L, the imaging ranges 60LL and 60LR of the imaging lenses 6LL and 6LR are overlapped and shifted according to the base line length and the height h. Similarly for the stereo camera 6R, the imaging ranges 60RL and 60RR by the imaging lenses 6RL and 6RR are shifted and overlapped according to the baseline length and the height h.

  Hereinafter, unless otherwise specified, the imaging ranges 60LL and 60LR and the imaging ranges 60RL and 60RR are collectively referred to as stereo imaging ranges 60L and 60R, respectively. As shown in FIG. 6B, the stereo cameras 6L and 6R are configured such that the stereo imaging ranges 60L and 60R include one end of the stereo imaging range 60L in the width direction of the vehicle 1 and the width of the stereo imaging range 60R in the width direction. The ends are arranged so as to overlap in the area 61 at a predetermined overlapping rate.

  The definition of the width direction includes not only the width direction of the vehicle 1 but also the width direction of the road (traveling surface), the width direction of the road (traveling surface), or the direction orthogonal to the traveling direction of the vehicle 1. In the definition of the width direction, it does not matter whether it is a straight road extending straight or a road curving to the left and right in the traveling direction. In this specification, the width direction of the road and the width direction of the road are, for example, the width between two opposing sidewalks on both sides of the road, and the width (lane width) between two opposing white lines on both sides of the road. It can be defined.

  FIG. 7 is a diagram illustrating the overlap in the traveling direction of the vehicle 1 of the stereo imaging range 60 </ b> L by the stereo camera 6 </ b> L that can be applied to the first embodiment. In FIG. 7, the left camera view VL and the right camera view VR respectively indicate the views (imaging ranges) of the imaging lenses 6LL and 6LR in the stereo camera 6L. It is assumed that the moving vehicle 1 has taken images twice with the stereo camera 6L. The stereo camera 6L captures an image of the stereo image capturing range 60L (the image capturing ranges 60LL and 60LR) for the first time, and the second time, the moving distance of the vehicle 1 in the traveling direction of the vehicle 1 with respect to the stereo image capturing range 60L. The stereo imaging range 60Ln (imaging ranges 60LLn and 60LRn) moved by the corresponding distance is imaged.

  When the stereo camera 6L performs the imaging operation twice while the vehicle 1 is moving in the traveling direction, the imaging timing of the stereo camera 6L is such that the stereo imaging range 60L and the stereo imaging range 60Ln are within the traveling direction visual field Vp. Is controlled so as to overlap by the traveling direction overlapping range Dr. The traveling direction overlap range may be set in advance. In this specification, the traveling direction overlapping range Dr is an overlapping range of a plurality of images in the traveling direction, and means a ratio of the size of the overlapping range to the size of one captured image (that is, the size of the visual field Vp). The ratio is called the traveling direction overlap rate. In order to simplify the expression, in the following, the size of the visual field Vp is set as a unit length, the overlapping range in the traveling direction is made equal to the overlapping rate in the traveling direction, and this overlapping rate is expressed by “Dr”. Used in (4). According to this configuration, by sequentially capturing images in the stereo imaging ranges 60L and 60Ln along the time axis, two stereo images can be obtained by capturing images in the stereo imaging ranges 60L and 60Ln. Two stereo images can be easily stitched together. Therefore, a plurality of stereo images used for road inspection are acquired by sequentially capturing images along the time axis using a stereo camera while the vehicle 1 travels or moves on the road surface 4 along the traveling direction. Can.

  Although the image processing system according to the first embodiment has been described above as using two stereo cameras 6L and 6R, the present invention is not limited to this example. For example, in the image processing system according to the first embodiment, as shown in FIG. 8A, one stereo camera 6C is further added to the stereo cameras 6L and 6R, and three stereo cameras 6L, You may comprise using 6R and 6C. Further, it is possible to configure using four or more stereo cameras such as four or five.

  In the example of FIG. 8 (a), the distance between the stereo cameras 6L and 6R is wider than when only the stereo cameras 6L and 6R are used, and the stereo camera 6C is arranged at the center thereof. As shown in FIG. 8B, the stereo imaging range 60C is configured by the imaging ranges 60CL and 60CR by the imaging lenses 6CL and 6CR of the stereo camera 6C. Stereo cameras 6L, 6C, and 6R are arranged such that the respective stereo imaging ranges 60L, 60C, and 60R overlap with each other in the width direction of vehicle 1 at a predetermined overlap rate.

  Thus, by using the three stereo cameras 6L, 6C and 6R to image one lane, it is possible to set the imaging range on the right side, the center and the left side of the lane, and to capture images with high image quality ( A high-resolution stereo image can be captured with a small number of stereo cameras. Here, in particular, the road width is generally defined as 3.5 m. Therefore, in correspondence with this road width of 3.5 m, it is conceivable that both ends of the lane in the road width direction are imaged by the stereo cameras 6L and 6R, and the center portion is imaged by the stereo camera 6C. The overlapping ranges 60L, 60C, and 60R at least span the lane width, and all the width directions of one lane of the traveling surface are imaged. Even when four or more stereo cameras are used, imaging can be performed by setting the imaging range.

  The crack rate and flatness need to connect the stereo imaged images in the moving body traveling direction (extension direction of the road), but are not necessarily required for measuring the rutting amount. In other words, since the measurement of rutting is the measurement object in the width direction of the moving body (the width direction of the road), there is no need to connect stereo images in the traveling direction of the moving body, and there are multiple stereo cameras in the width direction of the road. It is necessary to connect stereo images.

  For example, it is an agreement that we measure five locations on a 100m section of the road. In this case, stereo imaging is performed at arbitrary five locations (no foreign matter such as manholes are present), and each stereo image is connected in the width direction to create one depth map, so that the depths D1 and D2 are You can get The rutting amount D can be measured based on the acquired D1 and D2 and the cross-sectional information of the depth map. When measuring only the amount of rutting, there is no need to connect a stereo image in the moving direction of the moving body or to create a depth map in the moving direction, so the data capacity can be reduced (save). The image processing time can be shortened.

  The image processing system may be configured using only one stereo camera having an angle of view that can cover the road width (3.5 m) where the imaging range is defined.

Configuration of Image Processing System According to First Embodiment
Next, the configuration of the image processing system according to the first embodiment will be described. In the following description, it is assumed that the image processing system includes two stereo cameras.

  FIG. 9 is a block diagram illustrating an example of a schematic configuration of the image processing system according to the first embodiment. 9, the image processing system 10 includes imaging units 100-1 and 100-2, imaging control units 101-1 and 101-2, a speed acquisition device 102, and a generation unit 103.

  The imaging units 100-1 and 100-2 correspond to the stereo cameras 6L and 6R described above, respectively. The imaging control units 101-1 and 101-2 control imaging operations such as imaging timing, exposure, and shutter speed of the imaging units 100-1 and 100-2, respectively. The speed acquisition device 102 acquires the speed of the imaging units 100-1 and 100-2 with respect to the subject (road surface 4). The generation unit 103 generates a trigger (imaging trigger) for designating imaging by the imaging units 100-1 and 100-2 based on the speed acquired by the speed acquisition device 102 and the traveling direction visual field Vp. The generation unit 103 sends the generated trigger to the imaging control units 101-1 and 101-2. The imaging control units 101-1 and 101-2 cause the imaging units 100-1 and 100-2 to execute an imaging operation in response to the trigger sent from the generation unit 103.

  Further, a device having at least an imaging unit 100-1, an imaging unit 100-2 (that is, stereo cameras 6L and 6R), an imaging control unit 101, and an image processing unit to be described later is referred to as an imaging device 10A. FIG. 10 is a functional block diagram illustrating an example of the function of the imaging apparatus 10A. In FIG. 10, the imaging device 10A includes at least two or more imaging units 100-1 and 100-2, an imaging control unit 101 that controls the imaging operation of each imaging unit, and an image processing unit (image processing unit) 111. An output unit 104, a recording unit 107, an operation unit 108, and a wireless communication unit 110 are included. The image processing unit (image processing unit) 111 includes a matching processing unit 105 (corresponding to a matching processing unit 510 described later) and a 3D information generating unit 106 (corresponding to a 3D information generating unit 511 described later). Details of the image processing unit (image processing means) 111 will be described later.

  The recording unit 107 records an image captured by each imaging unit and an image subjected to a connecting process described later. The output unit 104 outputs the recorded image. Specifically, an image is written to an external recording unit such as an SD card or a CF card, or is output to an external device (for example, a PC, a server, or the like) through various communication cables. In addition, the output unit 104 can also output an image to the outside using wireless communication via the wireless communication unit 110.

  The imaging control unit 101 may be configured by the imaging control units 101-1 and 101-2 as shown in FIG. 9, and may cause the imaging units 100-1 and 100-2 to perform imaging operations.

  Further, the imaging units 100-1 and 100-2 are not limited to two, and may be configured of three, four, five, or more. In this case, the imaging control unit 101 corresponding to each imaging unit may be provided.

  FIG. 11 is a block diagram illustrating an example of a hardware configuration of the image processing system 10 according to the first embodiment. 11, the image processing system 10 includes stereo cameras 6L and 6R, and an information processing apparatus 50 corresponding to the PC 5 in FIG. The information processing device 50 generates a trigger at a predetermined timing, and sends the generated trigger to the stereo cameras 6L and 6R. The stereo cameras 6L and 6R perform imaging in response to this trigger. The respective stereo captured images captured by the stereo cameras 6L and 6R are supplied to the information processing apparatus 50. The information processing device 50 stores and accumulates each stereo captured image supplied from the stereo cameras 6L and 6R in a storage or the like. The information processing apparatus 50 performs image processing such as generation of a depth map and joining of the generated depth maps based on the accumulated stereo captured image.

  FIG. 12 is a block diagram illustrating an exemplary configuration of the stereo camera 6L according to the first embodiment. The stereo camera 6R can be realized with the same configuration as that of the stereo camera 6L, and thus the description thereof is omitted here.

  12, stereo camera 6L includes imaging optical systems 600L and 600R, imaging elements 601L and 601R, driving units 602L and 602R, signal processing units 603L and 603R, and an output unit 604. Among these components, the imaging optical system 600L, the imaging element 601L, the driving unit 602L, and the signal processing unit 603L correspond to the imaging lens 6LL described above. Similarly, the imaging optical system 600R, the imaging element 601R, the drive unit 602R, and the signal processing unit 603R are configured to correspond to the imaging lens 6LR described above.

  The imaging optical system 600L is an optical system having an angle of view α and a focal length f, and projects light from the subject onto the imaging element 601L. The image sensor 601L is an optical sensor using, for example, CMOS (Complementary Metal Oxide Semiconductor), and outputs a signal corresponding to the projected light. Note that an optical sensor using a CCD (Charge Coupled Device) may be applied to the image sensor 601L. The drive unit 602L drives the image sensor 601L, performs predetermined processing such as noise removal and gain adjustment on the signal output from the image sensor 601L, and outputs the signal. The signal processing unit 603L performs A / D conversion on the signal output from the driving unit 602L, and converts the signal into a digital image signal (captured image). The signal processing unit 603L performs predetermined image processing such as gamma correction on the converted image signal and outputs it. The captured image output from the signal processing unit 603L is supplied to the output unit 604.

  The operations of the imaging optical system 600R, the imaging element 601R, the driving unit 602R, and the signal processing unit 603603R are the same as those of the imaging optical system 600L, the imaging element 601L, the driving unit 602L, and the signal processing unit 603L. Description of is omitted.

  For example, the triggers output from the information processing apparatus 50 are supplied to the drive units 602L and 602R. The drive units 602L and 602R take signals from the image sensors 601L and 601R according to the trigger and perform imaging.

  Here, the drive units 602L and 602R perform exposure in the image sensors 601L and 601R by a simultaneous simultaneous exposure method.

  By the way, in this embodiment, since imaging is performed while the vehicle 1 is traveling, the captured subject (road surface) may be distorted. This problem appears more prominently in the line exposure sequential readout system (so-called rolling shutter system) in which readout is sequentially performed for one line to a plurality of lines of pixels of the imaging device. This is because this is a readout method in which light is taken in order (line order) from the top of the pixel position, so that each line in the frame is not a photograph of the subject at exactly the same time. In the case of the rolling shutter system, if the camera or the subject moves at high speed while capturing an image signal of one frame, the image of the subject is captured with a shift depending on the line position. It is difficult to obtain accurate distance information from such a captured image.

  Therefore, the image pickup devices 601L and 601R of the stereo cameras 6L and 6R of the present embodiment employ a simultaneous exposure batch reading method (so-called global shutter method) that reads all pixels at the same timing. As a result, even if the image is taken while the vehicle 1 is traveling, the road surface shape can be projected geometrically correct without distortion of the road surface, and accurate distance information can be obtained.

  The output unit 604 outputs the captured images of each frame supplied from the signal processing units 603L and 603R as a set of stereo captured images. The stereo captured image output from the output unit 604 is sent to the information processing apparatus 50 and accumulated.

  FIG. 13 is a block diagram illustrating a configuration of an example of the information processing apparatus 50 applicable to the first embodiment. In FIG. 13, an information processing apparatus 50 includes a CPU (Central Processing Unit) 5000, a ROM (Read Only Memory) 5001, a RAM (Random Access Memory) 5002, and a graphics I / F (interface) connected to a bus 5030, respectively. ), A storage 5004, an input device 5005, a data I / F 5006, and a communication I / F 5007. Further, the information processing device 50 includes a camera I / F 5010a and a speed acquisition device 5021 that are connected to the bus 5030, respectively.

  The storage 5004 is a storage medium that stores data in a nonvolatile manner, and a hard disk drive or a flash memory can be applied. The storage 5004 stores programs and data for the CPU 5000 to operate.

  The CPU 5000 controls the overall operation of the information processing apparatus 50 using the RAM 5002 as a work memory, for example, according to a program stored in advance in the ROM 5001 or the storage 5004. The graphics I / F 5003 generates a display signal that can be supported by the display 5020 based on the display control signal generated by the CPU 5000 according to the program. The display 5020 displays a screen corresponding to the display signal supplied from the graphics I / F 5003.

  The input device 5005 receives a user operation and outputs a control signal corresponding to the received user operation. As the input device 5005, a pointing device such as a mouse or a tablet or a keyboard can be applied. Alternatively, the input device 5005 and the display 5020 may be integrally formed to have a so-called touch panel configuration.

  A data I / F 5006 exchanges data with an external device. As the data I / F 5006, for example, USB (Universal Serial Bus) can be applied. A communication I / F 5007 controls communication with an external network in accordance with an instruction from the CPU 5000.

  The camera I / F 5010a is an interface to each of the stereo cameras 6L and 6R. Each stereo image output from each stereo camera 6L and 6R is passed to, for example, the CPU 5000 via the camera I / F 5010a. Also, the camera I / F 5010a generates the above-described trigger according to the instruction of the CPU 5000, and sends the generated trigger to each of the stereo cameras 6L and 6R.

  The speed acquisition device 5021 acquires speed information indicating the speed of the vehicle 1. When the stereo cameras 6L and 6R are attached to the vehicle 1, the speed information acquired by the speed acquisition device 5021 indicates the speed of the stereo cameras 6L and 6R with respect to the subject (road surface). The speed acquisition device 5021 has, for example, a function of receiving a GNSS (Global Navigation Satellite System) signal, and acquires speed information indicating the speed of the vehicle 1 based on the Doppler effect of the received GNSS signal. Not limited to this, the speed acquisition device 5021 can also acquire speed information directly from the vehicle 1.

  FIG. 14 is a functional block diagram illustrating an example of functions of the information processing apparatus 50 that can be applied to the first embodiment. 14, the information processing apparatus 50 includes a captured image acquisition unit 500, a UI unit 501, a control unit 502, an imaging control unit 503, and an image processing unit (image processing unit) 523. The information processing apparatus 50 further includes a 3D information acquisition unit 520, a state characteristic value calculation unit 521, and a record creation unit 522. The image processing unit (image processing unit) 523 includes a matching processing unit 510 and a 3D information generation unit 511.

  The captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, and record creation unit 522 This is realized by a program operating on the CPU 5000. Without being limited thereto, the captured image acquisition unit 500, UI unit 501, control unit 502, imaging control unit 503, matching processing unit 510, 3D information generation unit 511, 3D information acquisition unit 520, state characteristic value calculation unit 521, and records A part or all of the creation unit 522 may be configured by a hardware circuit that operates in cooperation with each other.

  The captured image acquisition unit 500 functions as an acquisition unit, and acquires stereo captured images from the stereo cameras 6L and 6R. The captured image acquisition unit 500 stores the acquired stereo captured image in the storage 5004, for example. The captured image acquisition unit 500 acquires a stored stereo captured image from the storage 5004, for example.

  The UI unit 501 realizes a user interface by display on the input device 5005 and the display 5020. The control unit 502 controls the overall operation of the information processing apparatus 50.

  The imaging control unit 503 functions as a control unit, and corresponds to the imaging control units 101-1 and 101-2, the speed acquisition device 102, and the generation unit 103 described above. That is, the imaging control unit 503 acquires speed information indicating the speed of the stereo cameras 6L and 6R with respect to the subject (road surface 4), and acquires the acquired speed information and a preset angle of view α of the stereo cameras 6L and 6R. Based on the height h, a trigger for instructing imaging of each stereo camera 6L and 6R is generated.

  The matching processing unit 510 functions as an image processing unit, and performs a matching process using two captured images constituting the stereo captured image acquired by the captured image acquisition unit 500. The 3D information generation unit 511 functions as an image processing unit, and performs processing related to three-dimensional information. For example, the 3D information generation unit 511 obtains depth information by trigonometry or the like using the result of the matching processing by the matching processing unit 510, and generates 3D point cloud information based on the obtained depth information.

  Here, an example of the image processing unit (image processing means) 111 will be described. The image processing means generates a depth map in which the captured images acquired by the captured image acquisition unit 500 are arranged like an image by depth matching corresponding to each pixel by stereo matching for each captured stereo camera. When the PC 5 has an image processing unit, a plurality of stereo captured images are acquired, and the PC 5 generates a depth map. In that case, the positional relationship of the captured image can be grasped by recording information on which stereo camera is captured in the stereo captured image, and an imaging region overlapping with an adjacent stereo camera, which will be described later, can be derived. When the stereo camera 6 has image processing means, the stereo camera 6 may generate a depth map, and the PC 5 may acquire the generated depth map. Furthermore, when both the stereo camera 6 and the PC 5 have image processing means, it is possible to appropriately select which image processing means generates the depth map.

  The matching processing unit 510 obtains the relative positions of the stereo cameras so that the depth distances of the images corresponding to the imaging regions overlapping with the adjacent stereo cameras correspond to the plurality of depth maps (that is, the overlapping imaging regions). The position where they are connected is determined).

  A specific description of the image processing means is as follows.

  The matching processing unit 510 aligns the plurality of depth maps. In alignment, it is determined whether or not the relative positions of locations corresponding to overlapping imaging regions in a plurality of depth maps match or do not match. At this time, if the relative position is obtained so that the luminance of the pixel also corresponds, the accuracy of alignment is improved. ICP (Iterative Closest Point) can be used as a matching method using a depth map. In addition, SfM can be used as a matching method using a luminance image.

  Then, the matching processing unit 510 determines a connecting position at a position where the relative position matches in the plurality of depth maps. Further, the matching processing unit 510 performs a combining process for combining a plurality of depth maps (images) for which connection positions are determined into one depth map (image) (also referred to as a connecting process).

  The 3D information generation unit 511 generates three-dimensional point group information for the combined depth map (image).

  In addition, although the above-mentioned description was demonstrated with "the image of the imaging region which overlaps with an adjacent stereo camera", the matching process of "the image of the imaging region which overlaps with the advancing direction of a vehicle" is also the same.

  When measuring the rutting amount, distance information in the width direction of the road (traveling surface) is necessary, and therefore matching processing of “images of imaging regions overlapping with adjacent stereo cameras” is performed. On the other hand, when the flatness is to be measured, distance information on the traveling direction of the vehicle on the road is necessary, and matching processing of “an image of an imaging area overlapping in the traveling direction of the vehicle” is performed.

  Note that, as an example, a plurality of depth maps are combined (process for connecting), but only luminance images may be combined (process for connecting) similarly. Note that the luminance image is obtained, for example, by capturing an image with only one of the stereo cameras (6LL or 6LR only in the case of the stereo camera 6L). In this case, it is possible to acquire images of imaging regions that overlap at least in the vehicle traveling direction.

  The 3D information acquisition unit 520 functions as a measurement unit, and acquires 3D point group information obtained for each stereo captured image by the 3D information generation unit 511. The state characteristic value calculation unit 521 functions as a measurement unit, and each 3D point group information acquired by the 3D information acquisition unit 520 and each stereo captured image acquired by the captured image acquisition unit 500. Using these, the state characteristic values of crack rate C, rutting amount D, and flatness σ for calculating MCI are calculated. The record preparation unit 522 calculates MCI based on each state characteristic value calculated by the state characteristic value calculation unit 521, and prepares a record.

  A program for realizing each function according to the first embodiment in the information processing apparatus 50 is a file in an installable format or an executable format, which is a CD (Compact Disk), a flexible disk (FD), a DVD (Digital Versatile). Provided on a computer-readable recording medium such as a disk). However, the present invention is not limited to this, and the program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. In addition, the program may be provided or distributed via a network such as the Internet.

  The program includes a captured image acquisition unit 500, a UI unit 501, a control unit 502, an imaging control unit 503, a matching processing unit 510, a 3D information generation unit 511, a 3D information acquisition unit 520, a state characteristic value calculation unit 521, and a record creation unit. The module configuration includes 522. As actual hardware, the CPU 5000 reads the program from a storage medium such as the storage 5004 and executes it, whereby the above-described units are loaded onto a main storage device such as the RAM 5002, and the captured image acquisition unit 500 and UI unit 501. The control unit 502, the imaging control unit 503, the matching processing unit 510, the 3D information generation unit 511, the 3D information acquisition unit 520, the state characteristic value calculation unit 521, and the record creation unit 522 are generated on the main storage device. ing.

[Trigger generation method according to the first embodiment]
Next, a method of generating a trigger for instructing each of the stereo cameras 6L and 6R to capture an image according to the first embodiment will be described in more detail. In the first embodiment, the generation unit 103 moves the distance in the traveling direction of the vehicle 1 within the stereo imaging range of the stereo cameras 6L and 6R with respect to the measurement object (road surface 4) in the time when the vehicle 1 moves according to the speed information. In contrast, the imaging trigger is generated at a time interval shorter than the time.

  That is, the trigger needs to be generated so that the photographing range of the road surface 4 in the stereo cameras 6L and 6R maintains a predetermined overlapping rate (traveling direction overlapping rate Dr) in the traveling direction of the vehicle 1. The purpose of this is to enable detection of sufficient corresponding points in order to stably calculate a highly accurate camera position in the process of calculating the imaging position from each stereo captured image, as will be described later. The lower limit value of the traveling direction overlap rate Dr is determined experimentally, for example, as “60%”. In this case, stereo imaging is performed such that the traveling direction overlap rate Dr is 60% or more (Dr ≧ 60%).

In the first embodiment, the following three methods can be applied as a trigger generation method.
(1) Method of generating at regular time intervals (first generation method)
(2) Method of generating by detecting the moving speed of the camera (second generation method)
(3) Method of calculating and generating moving distance using captured image (third generation method)

(First generation method)
First, a first trigger generation method will be described. In the first generation method, the trigger time interval is determined from the maximum speed Speed of the vehicle 1 being imaged and the size of the imaging range (the length of the imaging range in the traveling direction of the vehicle 1). The speed acquisition device 5021 acquires in advance the maximum speed Speed of the vehicle 1 based on a system setting value in the vehicle 1 or a user input to the information processing device 50. The imaging control unit 503 obtains the maximum speed Speed from the speed acquisition device 5021, and uses the following formula to determine the trigger time interval, the acquired maximum speed Speed, the traveling direction visual field Vp, and the traveling direction overlap rate Dr. Calculate by (4). The lower limit value described above is applied to the traveling direction overlap rate Dr.

  The number of triggers fps to be generated in one second is calculated by equation (4). The reciprocal of the trigger number fps is the time interval until the next trigger to be generated.

  The traveling direction visual field Vp is schematically represented by the height h from the road surface 4 of each stereo camera 6L and 6R and the angle of view α of each stereo camera 6L and 6R, as described with reference to FIG. Can be set based on. Actually, the traveling direction visual field Vp is set in consideration of the angles of the stereo cameras 6L and 6R with respect to the road surface 4 and the like.

  Here, when the moving vehicle 1 curves to the right or left, the movement amount of the camera outside the stereo cameras 6L and 6R increases. Therefore, it is more preferable to use the speed of rotational movement based on the position of the outside camera instead of using the maximum speed Speed of the vehicle 1 as it is.

  The imaging control unit 503 stores and accumulates all of the stereo captured images captured according to the generated trigger, for example, in the storage 5004 or the RAM 5002.

(Second generation method)
Next, the second trigger generation method will be described. While the first method described above is simple, in the state in which the vehicle 1 is stopped or in the state in which the vehicle 1 is moving at a lower speed than a predetermined speed, imaging is performed at excessively fine intervals with respect to the maximum speed Speed. As a result, the amount of stereo images to be stored increases. In the second generation method, the imaging control unit 503 detects the moving speed of the camera, and generates a trigger according to the detected moving speed.

  The imaging control unit 503 calculates the time interval until the next trigger using the current speed of the vehicle 1 indicated by the speed information acquired by the speed acquisition device 5021 as the maximum speed Speed of Expression (4), and performs imaging. I do. According to the second generation method, as the moving speed of the vehicle 1 is smaller (slower), the time interval of trigger generation becomes longer, and unnecessary imaging is suppressed.

  The imaging control unit 503 stores and accumulates all of the stereo captured images captured according to the generated trigger, for example, in the storage 5004 or the RAM 5002.

  The second generation method and the first generation method described above can be implemented in combination.

(Third generation method)
Next, a third method of generating a trigger will be described. In the third generation method, a trigger is generated at regular time intervals based on the maximum speed Speed of the vehicle 1 as in the first generation method described above. Here, in the third generation method, not all of the stereo captured images captured in response to the trigger are accumulated, but only when the traveling direction overlap rate Dr falls below a preset value.

  Although mentioned later, it is possible to calculate the movement distance of a camera (vehicle 1) using only a stereo image pick-up picture picturized so that the direction of movement of vehicles 1 might overlap.

  The imaging control unit 503 calculates the moving distance of the camera using the last captured stereo captured image and the stereo captured image captured according to the immediately preceding (one time before) imaging trigger. The imaging control unit 503 determines whether the calculated movement distance of the camera exceeds the movement distance (movement distance threshold) corresponding to the lower limit value of the traveling direction overlap rate Dr. When it is determined that the calculated movement distance of the camera exceeds the movement distance threshold value, the imaging control unit 503 accumulates a stereo image taken immediately before. On the other hand, when it is determined that the calculated movement distance of the camera does not exceed the movement distance threshold value, the imaging control unit 503 discards the stereoscopic image captured immediately before. In other words, if the image overlap rate between the last captured stereo image and the last captured stereo image is equal to or less than the threshold value, the movement distance threshold is not exceeded. Discard the previously captured stereo image.

  Thus, useless image accumulation is not performed when the moving speed is low (slow) without using a sensor for measuring the current speed of the vehicle 1. In addition, useless image accumulation is not performed even when the moving speed is zero (not moving).

[Calculation method of road surface property value according to first embodiment]
Next, a road surface property value calculation method according to the first embodiment will be described. Below, the measuring method of flatness (sigma), rutting amount D, and the crack rate C which concerns on 1st Embodiment is demonstrated.

(Flatness)
FIG. 15 is a flowchart of an example illustrating a process of calculating flatness according to the first embodiment. In step S100, the captured image acquisition unit 500 acquires images captured by the stereo cameras 6L and 6R and stored in the storage 5004, for example. If a stereo picked-up image is acquired, a process will transfer to step S101a and step S101b which can be processed in parallel.

  In step S101a, the image processing unit 523 generates a depth map based on the stereo captured image acquired in step S100. Depth map generation processing applicable to the first embodiment will be described with reference to FIGS. 16 and 17. FIG. 16 is a flowchart illustrating an example of a depth map generation process applicable to the first embodiment.

  In step S120, the matching processing unit 510 acquires a stereo captured image from the captured image acquisition unit 500. In the next step S121, the matching processing unit 510 performs a matching process based on the two captured images constituting the acquired stereo captured image. In the next step S122, the 3D information generation unit 511 calculates depth information based on the matching processing result in step S121, and generates a depth map which is three-dimensional point group information.

  The processes of step S121 and step S122 will be described more specifically. In the first embodiment, depth information is calculated by a stereo method using two captured images constituting a stereo captured image. The stereo method here uses two captured images captured from different viewpoints by two cameras, and a pixel (reference pixel) in one captured image corresponds to a corresponding pixel ( This is a method of obtaining the corresponding pixel) and calculating the depth (depth distance) by trigonometry based on the reference pixel and the corresponding pixel.

  In step S121, the matching processing unit 510 uses the two captured images constituting the stereo captured image acquired from the captured image acquisition unit 500, and has a predetermined size centered on the reference pixel in one of the reference captured images. An area in the other captured image to be searched corresponding to the area is moved in the other captured image to perform the search.

  Various methods are known for searching for corresponding pixels. For example, a block matching method or an SGM (Semi-Global-Matching) propagation method can be applied.

  In the block matching method, a pixel value of an area cut out as a block of M pixels × N pixels around a reference pixel in one captured image is acquired. Also, in the other captured image, the pixel value of the area cut out as a block of M pixels × N pixels centering on the target pixel is acquired. Based on the pixel value, the similarity between the area including the reference pixel and the area including the target pixel is calculated. Similarities are compared while moving a block of M pixels × N pixels in the search target image, and the target pixel in the block with the highest similarity is set as a corresponding pixel corresponding to the reference pixel.

  Next, the SGM propagation method will be described. The matching processing unit 510 calculates the propagation cost Lr using an algorithm called SGM propagation method, and uses the propagation cost Lr (for example, the propagation cost L1) to calculate the energy cost S of the pixel p of interest. An energy calculation process for calculating p, d) is performed. The SGM propagation method is a form of dense algorithm. The energy cost S (p, d) of each pixel can be calculated using the following formulas (5) and (6).

  In equation (5), p represents the coordinates of the pixel 1100 and d represents the parallax. “P1” and “P2” are parameters frequently used by the SGM algorithm known as penalty value, and the values of P1 and P2 set to be different for each propagation direction are effectively used in the SGM propagation method can do.

  Based on the propagation cost from each direction calculated for each pixel, the energy cost S (p, d) of each pixel is calculated by the following equation (6).

The similarity can be calculated by various calculation methods. For example, the normalized cross-correlation (NCC) shown in the equation (7) is one of cost functions, and the higher the value of the numerical value C _NCC indicating the cost, the higher the similarity. Indicates. In equation (7), the values M and N represent the size of the pixel block for performing the search. Also, the value I (i, j) represents the pixel value of a pixel in a pixel block in one captured image serving as a reference, and the value T (i, j) represents a pixel block in the other captured image serving as a search target Represents the pixel value.

As described above, the matching processing unit 510 moves the pixel block in the other captured image corresponding to the pixel block of M pixels × N pixels in one captured image, for example, in pixel units in the other captured image. While performing the calculation of Equation (7), the numerical value C _NCC is calculated. In the other captured image, the center pixel of the pixel block having the maximum numerical value C _NCC is set as a target pixel corresponding to the reference pixel.

  As a method of calculating the similarity (correlation value), SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), ZNCC (Zero-mean Normalized Cross-Correlation), and the like can be given.

  SAD is a method of calculating the absolute value of the difference between luminance values and calculating the sum as shown in the following equation (8). According to SAD, the value decreases as the blocks are more similar.

  SSD is a method of calculating the sum of the squares of the luminance value differences as shown in the following equation (9). According to SSD, the value decreases as the blocks become more similar.

  ZNCC is a method of calculating a normalized cross-correlation after subtracting an average value as shown in the following formula (10).

  Returning to the description of FIG. 16, in step S122, the 3D information generation unit 511 calculates the depth distance (depth information) using trigonometry based on the reference pixels and the corresponding pixels obtained by the matching processing in step S121. And generating three-dimensional point cloud information relating to one captured image and the other captured image constituting the stereo captured image.

FIG. 17 is a diagram for explaining trigonometry applicable to the first embodiment. Processing to calculate the distance S to the target object 403 (for example, one point on the road surface 4) in the figure from the imaging position information in the image imaged by each imaging element 402 (corresponding to the imaging elements 601L and 601R) Is the purpose. That is, this distance S corresponds to the depth information of the target pixel.
The distance S is calculated by the following equation (11).

  In equation (11), the value baseline represents the length of the base line (base line length) between the cameras 400a and 400b. In the example of FIG. 6, this corresponds to the baseline length by the imaging lenses 6LL and 6LR (in the case of the stereo camera 6L). The value f represents the focal length of the lens 401 (corresponding to the imaging lenses 6LL and 6LR). The value q represents disparity. The parallax q is a value obtained by multiplying the pixel pitch of the imaging device by the difference between the coordinate values of the reference pixel and the corresponding pixel. The coordinate value of the corresponding pixel is obtained based on the result of the matching process in step S121.

  This equation (11) is a method for calculating the distance S when the two cameras 400a and 400b, that is, the imaging lenses 6LL and 6LR are used. This calculates the distance S from the captured images captured by the two cameras 400a and 400b, that is, the imaging lenses 6LL and 6LR, respectively. In the first embodiment, the calculation method according to this equation (11) is applied to the imaging lenses 6LL and 6LR of the stereo camera 6L and the respective captured images captured by the imaging lenses 6RL and 6RR of the stereo camera 6R, The distance S is calculated for each pixel.

  Returning to the description of FIG. 15, in step S101 b, the 3D information generation unit 511 estimates the camera position and orientation. Here, the camera position indicates, for example, the center coordinates when the stereo camera 6L is captured integrally. Details of the processing in step S101b will be described later.

  When the process of step S101a and step S101b is completed, the process proceeds to step S102. In step S102, the 3D information generation unit 511 calculates the previous depth map so that it matches the coordinate system of the previous time depth map based on the camera position and orientation estimated in step S101b for two depth maps adjacent in time series. The coordinate conversion of the depth map of the next time after the time is performed.

  The first and second imagings were performed for the first imaging and the second imaging that were sequentially performed on the time series according to the trigger in the stereo cameras 6L and 6R. The time is the previous time, and the time when the second imaging was performed is the next time. That is, the depth map at the previous time is a depth map generated based on the stereo image captured by the first imaging, and the depth map at the next time is a depth map generated based on the stereo image captured by the second imaging. is there.

  In the next step S103, the 3D information generation unit 511 integrates the depth map of the next time coordinate-converted in step S102 into the depth map of the previous time. In other words, since the two depth maps of the previous time and the next time are arranged in a common coordinate system by the coordinate conversion in step S102, these two depth maps can be integrated. The processes of step S102 and step S103 are performed on stereo captured images captured at all times.

  6 and 8, when a plurality of stereo cameras 6L and 6R or a plurality of stereo cameras 6L, 6C and 6R are used in the road width direction, stereo cameras arranged in the road width direction (for example, stereo) In the cameras 6L and 6R), the camera position and orientation are estimated in the same manner, and then depth map integration is performed.

  Further, the moving distance of the camera (vehicle 1) in the third generation method described above can be calculated in the coordinate transformation in step S102 and the depth map integration process in step S103.

  In the next step S104, the 3D information acquisition unit 520 acquires the depth map integrated in step S103 from the 3D information generation unit 511. Then, the state characteristic value calculation unit 521 calculates the flatness σ based on the depth map acquired by the 3D information acquisition unit 520. That is, when the depth maps are integrated, the road surface shape is restored as a point group in one three-dimensional space for the roads in the measured section. When the point cloud is restored, for each sampling point on the road surface, take the coordinate system connecting the coordinates of the 1.5m front and back points according to the regulations, and calculate the distance to the 3D point group in the center part, The displacement amount d can be calculated in the same manner as (2), and the flatness σ can be calculated from the displacement amount d according to the equation (3).

(Wad digging amount)
For the rutting amount D, in step S104, the state characteristic value calculation unit 521 scans the depth map acquired by the 3D information acquisition unit 520 in the road width direction, as shown in FIG. 2 (a) and FIG. 2 (b). Depths D1 and D2 can be obtained as shown in. The rutting amount D can be determined based on the acquired depths D1 and D2 and information of a cross section obtained by scanning the depth map.

  The measurement of the rutting amount D will be described in more detail.

  N (for example, 2 to 3) stereo cameras 6 capture images synchronously. More specifically, N (for example, 2 to 3) stereo cameras 6 suppress variations in transmission delay of imaging triggers (signals) in order to capture images synchronously.

  The captured image acquisition unit 500 acquires a plurality of captured stereo images.

  The image processing unit 523 generates a depth map in which depth distances corresponding to the respective pixels are arranged like an image by stereo matching for each stereo camera.

  The matching processing unit 510 determines the relative position of the stereo cameras (that is, the connecting position is determined) such that the depth distances of the images of the overlapping imaging area with the adjacent stereo cameras correspond to the plurality of depth maps. . At this time, it is further preferable to correspond not only to the depth distance but also to the luminance of the pixel. The matching processing unit 510 performs a composition process for compositing a plurality of depth maps (images) for which connection positions are determined so as to become one depth map (image).

  Next, the 3D information generation unit 511 generates one integrated point group data (distance information) from one combined depth map (image).

  Next, the state characteristic value calculation unit 521 projects the point cloud data (distance information) vertically above the road surface to generate a road surface image having the distance information. Here, FIG. 18 is a diagram illustrating an example of an image of the road surface 4. As shown in FIG. 18, the state characteristic value calculation unit 521 uses the average value of the depth distance Z of the point cloud data within a certain width (for example, 1 cm) as a cross section. At that time, the state characteristic value calculation unit 521 is a road surface such as a white line or an end of the road surface (traveling surface) 4 which is a road end (grass, soil, side groove, etc.) ) The foreign matter in 4 is detected and excluded from the measurement of the depth distance (that is, the rutting amount) so as not to be an illegal value.

  FIG. 19 is a diagram showing an example of measurement of rut value. FIG. 19 (a) shows the value of rutting amount, and FIG. 19 (b) shows a road surface image. As shown in FIG. 19 (a), the value of the rutting amount determines an estimated surface shape (that is, “base” in FIG. 19 (a)) with respect to the cross-sectional area to be inspected, and is the most convex shape from the estimated surface shape. Calculated by calculating the maximum value of the length of the extended line segment (which is perpendicular to the estimated surface shape) up to the point of (ie, the “value” in FIG. 19 (a)). As shown in FIGS. 2 (a) and 2 (b), the depth of the ruts was measured from the estimated surface shape (ie, the line extending across one lane in FIG. 2 (a)). It can be measured by drawing a line (perpendicular to the estimated surface shape) to the profile showing the value (ie, the thick line in FIG. 2 (b)). Accordingly, in FIG. 19A, the line extending to the most convex point is actually perpendicular to the estimated surface shape, but the scale ratio between the vertical axis and the horizontal axis in FIG. 18A is changed. Therefore, in FIG. 19 (a), the extension line actually appears to be slightly inclined with respect to the estimated surface shape of FIG. 19 (a). However, as described above, the line extending from the estimated surface shape in FIG. 19A is actually a line perpendicular to the estimated surface shape.

(Crack rate)
The state characteristic value calculation unit 521 applies, for example, the stereo captured image acquired in step S100 to the depth map integrated in step S103. That is, the state characteristic value calculation unit 521 integrates the respective stereo captured images that are captured in the traveling direction of the vehicle 1 with the traveling direction overlap rate Dr. The state characteristic value calculation unit 521 sets a mesh of 50 cm according to the regulations for the integrated image, acquires information such as cracks and patching in each mesh by image analysis, and the crack rate C according to the regulations based on the acquired information. Is calculated.

(Camera position and orientation estimation process)
Next, the camera position and orientation estimation process in step S101b in the flowchart of FIG. 15 described above will be described in more detail. FIG. 20 is a flowchart illustrating an example of camera position and orientation estimation processing applicable to the first embodiment. In the first embodiment, the camera position and orientation are estimated using SfM described above.

  In step S130, the 3D information generation unit 511 acquires a stereo captured image from the captured image acquisition unit 500. Here, the 3D information generation unit 511 acquires two stereo captured images that are continuously captured in time series (a stereo captured image at the previous time and a stereo captured image at the next time). Next, in step S131, the 3D information generation unit 511 extracts feature points from each acquired stereo image. This feature point extraction process is a process of finding a point that is easy to detect as a corresponding point of each stereo image, and typically, a point called a corner where there is a change in the image and the change is not uniform is detected. Do.

  In the next step S132, the 3D information generation unit 511 detects a point obtained by imaging the same location as the point extracted as the feature point in the stereo captured image at the previous time from the stereo captured image at the next time. For this detection process, a technique called optical flow or a technique called feature point matching represented by SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), or the like can be applied.

  In the next step S133, the 3D information generation unit 511 estimates the initial position and orientation of the camera. The 3D information generation unit 511 uses the coordinates of the corresponding points detected in each stereo captured image in step S132 as fixed values, and solves simultaneous equations using the position and orientation of the camera that captured the stereo captured image at the next time as parameters. Then, the position of the camera is estimated and three-dimensional coordinates are calculated (step S134).

A method for estimating the initial position and orientation of the camera will be described with reference to FIG. Expression (12) represents the relationship between coordinates x _{ij obtained} by projecting the point X _j of the space onto the camera (viewpoint) P _i . In Expression (12), the value n represents the number of points in the three-dimensional space, and the value m represents the number of cameras (captured images).

Each value P is a projection matrix for converting the coordinates of a point in the three-dimensional space into the two-dimensional coordinates of the image for each camera, and is represented by the following equation (13). Expression (13) is composed of a 2-by-2 transformation matrix of three-dimensional coordinates shown on the right side, and a projective transformation f _i from three-dimensional coordinates to two-dimensional coordinates.

In FIG. 21, the value X _j and the value P _i are calculated using simultaneous equations given only the coordinates x _ij captured by the camera. A linear least squares method can be used to solve this simultaneous equation.

  In the next step S135, the 3D information generation unit 511 optimizes the three-dimensional coordinates indicating the camera position calculated in step S134. The position and orientation of the camera calculated by solving the simultaneous equations may not have sufficient accuracy. Therefore, the accuracy is improved by performing the optimization process with the calculated value as the initial value.

Two-dimensional coordinates x _ij (re-projections calculated by the above equation (12) using the corresponding point coordinates acquired by the on-image search, the three-dimensional coordinates calculated in step S134 and the projection matrix of equation (13) as parameters The difference between the two is called the residual. Parameters are adjusted by an optimization operation so that the sum of the residuals is minimized for all corresponding points in all stereo captured images used for calculating the position and orientation of the camera. This is called bundle adjustment. By performing overall optimization through bundle adjustment, the accuracy of the position and orientation of the camera can be improved.

The above is the base processing for camera position and orientation estimation in normal SfM. In the first embodiment, since stereo cameras 6L and 6R whose base lengths are known are used, in addition to the detection of corresponding points in stereo captured images captured with overlapping imaging ranges according to the movement of the vehicle 1 Thus, it is easy to search for corresponding points in two captured images that constitute a stereo captured image. Therefore, the three-dimensional coordinate X _j in space can be determined with the actual scale (the actual size), and the position and orientation of the camera after movement can also be stably calculated.

  As described above, in the image processing system 10 according to the first embodiment, the information processing apparatus 50 generates a trigger and distributes the generated trigger to all the cameras 6L and 6R. The stereo cameras 6L and 6R are configured to perform imaging in synchronization. At this time, a clock generator included in each stereo camera 6L and 6R, a trigger distribution component for distributing triggers in the information processing apparatus 50, a difference in trigger wiring length in the information processing apparatus 50 and each stereo camera 6L and 6R, and the like Due to the influence, there is a possibility that a slight shift may occur in the timing at which each stereo camera 6L and 6R actually captures a trigger and executes imaging. It is preferable to suppress this shift in imaging timing as much as possible.

  In particular, it is desirable to suppress the deviation of the imaging timing as much as possible between the stereo cameras 6L and 6R whose base line length is known. For example, when wiring the trigger supply from the camera I / F 5010a to each of the stereo cameras 6L and 6R, it may be via a relay for securing the quality of the trigger or via a photocoupler for protection from electrostatic noise and the like. is there. In this case, it is desirable to share these repeaters and photocouplers between the stereo cameras 6L and 6R.

  In the image processing system 10 according to the first embodiment, the PC 5 serving as the information processing apparatus 50 is provided inside the vehicle 1 serving as the moving body, but the present invention is not limited to this. Here, FIG. 22 is a view showing another configuration example of the image processing system according to the first embodiment. As shown in FIG. 22, the image processing system 10 may be one in which a vehicle 1 as a moving body and a PC 5 as an information processing apparatus 50 are provided separately. Moreover, when sending to PC5, you may make it go through a cloud server. By storing an image in the cloud server, a plurality of PCs can refer to or download the same image, and a process of connecting the images with the plurality of PCs can be performed. Thereby, when inspecting the road property, it is possible to work from a plurality of bases, and the inspection efficiency is improved. In this case, for the connection with the PC, for example, when the imaging device and the vehicle 1 can communicate with each other, or when the imaging device and the vehicle 1 can communicate with each other, for example, wireless communication is performed between the vehicle 1 and the PC 5 that acquired the captured image. It is possible. In this case, for example, by adopting “5G”, which is a fifth generation mobile communication system that can provide data with higher speed, lower delay, larger capacity, and higher efficiency, communication with the cloud server is efficiently performed. be able to. Of course, conventional wireless communication can also be employed.

  In the image processing system 10, for example, the first business operator captures a plurality of images including distance information in the depth direction of the road (running surface), provides the captured images to the second business operator, It is also possible that the company of the company performs the above-mentioned image processing, outputs it as a report, and submits it to the administrative organization.

  In addition, the first business operator captures a plurality of images including distance information in the depth direction of the road (travel surface), provides the captured images to the second business operator, and the second business operator provides the above-described image. It is also possible to perform processing, provide the image after the image processing to a third business operator, and the third business operator outputs it as a record and submits it to an administrative institution.

Second Embodiment
Next, a second embodiment will be described.

  Hereinafter, in the description of the second embodiment, the description of the same parts as those of the first embodiment will be omitted, and different parts from the first embodiment will be described.

  In the first embodiment described above, a trigger for instructing the stereo cameras 6L and 6R to take an image is generated outside the stereo cameras 6L and 6R, for example, in the information processing apparatus 50, and is sent to the stereo cameras 6L and 6R. It was supplying. This is not limited to this example, and in the second embodiment, the trigger is generated by one of the stereo cameras 6L and 6R, thereby instructing the stereo cameras 6L and 6R to perform imaging.

  FIG. 23 is a block diagram showing an example of the hardware configuration of the image processing system 10 n according to the second embodiment. 23, image processing system 10n includes stereo cameras 6Ln and 6Rn and information processing apparatus 50n corresponding to stereo cameras 6L and 6R and information processing apparatus 50 in FIG. 11, respectively. One of the stereo cameras 6Ln and 6Rn (in this example, the stereo camera 6Ln) generates a trigger and instructs imaging by itself by the generated trigger. The stereo camera 6Ln supplies the generated trigger to the stereo camera 6Rn and instructs the stereo camera 6Rn to perform imaging.

  FIG. 24 is a block diagram showing an example of the configuration of a stereo camera 6Ln according to the second embodiment. In FIG. 24, the same parts as those in FIG. 12 described above are indicated by the same reference numerals and the detailed description will be omitted.

  Similarly to the stereo camera 6L described with reference to FIG. 12, the stereo camera 6Ln includes imaging optical systems 600L and 600R, imaging elements 601L and 601R, driving units 602L and 602R, signal processing units 603L and 603R, and an output unit. And 604. Stereo camera 6Ln further includes a control unit 610, a speed acquisition device 611, and a camera I / F 612.

  The speed acquisition device 611 corresponds to the speed acquisition device 5021 described with reference to FIG. 13, and acquires speed information indicating the speed of the stereo camera 6Ln. The speed acquisition device 611 has a function of receiving, for example, a GNSS signal, and acquires speed information indicating the speed of the stereo camera 6Ln based on the Doppler effect of the received GNSS signal. Not limited to this, the speed acquisition device 611 may acquire speed information indicating the speed of the vehicle 1 from the system of the mounted vehicle 1.

  Based on the speed information acquired by the speed acquisition device 611 and the imaging range of the stereo camera 6Ln, the control unit 610 generates a trigger for instructing the stereo camera 6Ln and the stereo camera 6Rn to perform imaging. As a trigger generation method, the first generation method or the second generation method described above can be applied. Further, control unit 610 prestores information on the imaging range of stereo camera 6Ln.

  The camera I / F 612 is an interface for the stereo camera 6Rn. The control unit 610 supplies the generated trigger to the drive units 602L and 602R and also supplies the stereo camera 6Rn via the camera I / F 612.

  The stereo camera 6Rn has a configuration equivalent to that of the stereo camera 6Ln. The trigger supplied from the stereo camera 6Ln is supplied to the camera I / F 612 included in the stereo camera 6Rn, and is supplied from the camera I / F 612 to the drive units 602L and 602R included in the stereo camera 6Rn. The stereo camera 6Rn on the side to which the trigger is supplied can omit the control unit 610 and the speed acquisition device 611.

  FIG. 25 is a block diagram showing an example of the configuration of an information processing apparatus 50 n applicable to the second embodiment. In FIG. 25, portions common to FIG. 13 will be assigned the same reference numerals and detailed descriptions thereof will be omitted. As illustrated in FIG. 25, the information processing device 50n has a configuration in which the speed acquisition device 5021 is omitted from the information processing device 50 illustrated in FIG. The camera I / F 5010b receives stereo captured images from the stereo cameras 6Ln and 6Rn and does not output a trigger. Further, as a function of the information processing device 50n, the imaging control unit 503 is omitted from the functional block diagram shown in FIG.

  Also in the image processing system 10n according to the second embodiment, similarly to the image processing system 10 according to the first embodiment described above, in the traveling direction of the vehicle 1 of stereo images taken by the stereo cameras 6Ln and 6Rn. It is possible to perform imaging with a predetermined traveling direction overlap rate Dr.

  Note that, when the stereo cameras 6Ln and 6Rn have the configuration shown in FIG. 24, the stereo cameras 6Ln and 6Rn may take an image in accordance with a trigger generated by the stereo cameras 6Ln and 6Rn. In this case, for example, clocks are shared by the stereo cameras 6Ln and 6Rn to synchronize imaging timing.

  Each of the above-described embodiments is a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the gist of the present invention.

  In each of the embodiments described above, the general road and the highway have been described as the traveling surface of the moving body, but the present invention can also be applied to a runway at an airfield. Further, it is possible to measure a hoistway (elevator shaft facing the elevator) such as a vertical hole-shaped reinforced concrete structure in which an elevator (elevator) that is a moving body travels. In particular, the inner wall of the hoistway may be deformed or cracked due to building deterioration due to deterioration over time or earthquakes. If this embodiment is applied, the inner wall of the hoistway can be measured by installing the stereo camera outside the elevator. Since the inside of the hoistway is dark and the luminance is insufficient at the time of imaging, it is necessary to appropriately install an illumination device such as an LED light source. Also, the materials of the roads and hoistways (such as asphalt and concrete) are arbitrary.

  The inner wall of the elevator hoistway is a facing surface that faces the outer surface of the elevator, and in this specification, the facing surface can also be defined as the traveling surface. Further, in the embodiment described above, the traveling surface and the opposing surface can be referred to as an inspection surface or an inspection target surface. Although the elevator does not travel on the wall of the hoistway, the embodiments described above contact a contact movable device (for example, a vehicle) or a surface (for example, a wall of an elevator shaft) moving on a road surface while contacting each other. It can be applied to a non-contact movable device (for example, an elevator) that moves smoothly. Therefore, the movable device is not limited to the contact-type movable device, and may be a non-contact-type movable device such as an aircraft or an unmanned aircraft. In the case of a non-contact movable device, it is important to maintain the relative position of the non-contact movable device with respect to the inspection surface (that is, not to change the relative position). However, if the position of the non-contact movable device can be accurately estimated, the change in the relative position of the non-contact movable device with respect to the inspection surface can be canceled (corrected). The change in position is not a problem for surface inspection. Similarly, the mobile device may be a device that is adapted to carry a human passenger or a device that does not carry a passenger (eg, a drone). In the latter case, the movable vehicle is remotely (ie, by a human operator or a computer system remote from the movable device) and / or automatically (ie by a computer system rather than a human operator). ) May be controlled. For example, automatic control such as “continue to travel to a position 1 m right from the white line on the left side of the lane” (ie, automatic driving) can reduce human operation errors of the vehicle and obtain a captured image of a stable traveling surface. Be

  In each of the above-described embodiments, a stereo camera is employed as a means for capturing a stereo image, but the present invention is not necessarily limited to a stereo camera. For example, it may be a monocular lens camera. In this case, there is a method of stopping the vehicle and imaging a plurality of times in the width direction of the traveling surface. In addition, there is an example in which a color filter is attached to a lens portion of a monocular lens camera. The color filter generates a color shift according to the distance to the subject, and the distance can be measured for each pixel by analyzing the color shift. As described above, the imaging means is not limited to a stereo camera as long as it is an imaging unit that can obtain a captured image from which “distance information in the depth direction” defined in this specification can be obtained.

  The image processing system may be an output system including an output unit that outputs an image subjected to the connecting process, instead of the image processing unit that performs the connecting process. For example, an acquisition unit for acquiring a plurality of captured images including distance information in the depth direction obtained by imaging the traveling surface of a moving object, and one image obtained by combining the acquired plurality of captured images in the width direction of the traveling surface The output system may include an output unit. Further, instead of the matching process, a plurality of captured images may be aligned using a learning model that has learned a large number of captured images.

  Here, an information processing method in the image processing unit (image processing means) 111 will be described. The image processing unit 111 generates a depth map in which the depth distances corresponding to the respective pixels are arranged like an image by stereo matching for each captured stereo camera for the captured image acquired by the captured image acquisition unit 500. This will be described in more detail with reference to the flowchart shown below.

  Here, FIG. 26 is a flowchart showing a flow of information processing in the image processing unit 111. As illustrated in FIG. 26, the image processing unit 111 acquires a plurality of captured images including distance information in the depth direction in which the traveling surface of the vehicle 1 is imaged by the stereo camera 6 (step S1). Next, the image processing unit 111 overlaps the acquired captured images and connects them in the width direction of the traveling surface (step S2).

  The connecting process in step S2 will be described in detail. FIG. 27 is a flowchart showing the flow of processing in step S2. As illustrated in FIG. 27, the image processing unit 111 determines a position where the relative positions of the plurality of captured images are connected (step S <b> 21), and the plurality of captured images are determined as one image at the determined connection positions. (Step S22), and three-dimensional point cloud information is generated from the combined image (step S23).

  The above embodiment may be configured as follows.

[Supplementary Note 1] An imaging device for imaging the shape of an object to be measured,
An imaging control unit that performs stereo imaging of the object to be measured in response to an imaging trigger and outputs a stereo captured image including a predetermined imaging range;
An acquisition unit configured to acquire velocity information indicating a velocity of the imaging device relative to the object;
A generation unit that generates the imaging trigger at a time interval shorter than the time for moving the distance in the speed direction in the imaging range with respect to the measurement object at a speed indicated by the speed information;
An imaging device comprising:

[Supplementary Note 2] The generation unit includes:
Based on the speed information and the distance, the stereo imaging image output from the imaging control unit in response to the imaging trigger, and the imaging in response to the imaging trigger one time before the imaging trigger The imaging apparatus according to appendix 1, wherein the imaging trigger is generated at the time interval at which an overlapping rate with the stereo captured image output from the control unit is equal to or greater than a threshold value.

[Supplementary Note 3] The acquisition unit
The imaging device according to Appendix 1 or 2, wherein the velocity information indicating the velocity predetermined as the fastest velocity of the imaging device with respect to the measurement object is acquired.

[Supplementary Note 4] The imaging control unit
The stereo-captured image output from the imaging control unit in response to the imaging trigger, and the stereo-captured image output from the imaging control unit in response to the imaging trigger immediately before the imaging trigger; The imaging apparatus according to supplementary note 3, wherein the stereo captured image output from the imaging unit is discarded in response to the imaging trigger of the previous time when the overlap rate of the image is equal to or less than a threshold value.

[Supplementary Note 5] The acquisition unit includes:
The imaging apparatus according to appendix 1 or appendix 2, which acquires the speed information indicating the speed of the imaging apparatus that is moving with respect to the measurement object.

[Supplementary Note 6] The generation unit
An imaging control unit provided with the imaging device and the other imaging device provided with the imaging trigger, the imaging range of the object to be measured is different from the direction of the speed of the imaging range of the imaging device The imaging device according to any one of appendices 1 to 5, which is supplied to the imaging control unit that performs the stereo imaging so as to partially overlap.

[Supplementary Note 7] A plurality of the imaging control units that each take a stereo image of the measurement object and output a stereo image including an imaging range corresponding to the stereo imaging,
The generator is
The imaging apparatus according to any one of supplementary notes 1 to 5, wherein the imaging trigger is supplied to each of the plurality of imaging control units.

[Supplementary Note 8] Each of the plurality of imaging control units includes:
The imaging apparatus according to appendix 7, wherein the corresponding imaging range is arranged so as to overlap a part of the adjacent imaging range in a direction different from the speed direction.

[Supplementary Note 9] An imaging method of an imaging device for imaging a shape of an object to be measured,
An imaging control step of performing stereo imaging of the object to be measured in response to an imaging trigger and outputting a stereo captured image including a predetermined imaging range;
An acquisition step of acquiring speed information indicating a speed of the imaging device with respect to the measurement object;
A generation step of generating the imaging trigger at a time interval shorter than the time for moving the distance in the speed direction in the imaging range with respect to the measurement object at a speed indicated by the speed information;
An imaging method comprising:

[Supplementary note 10] The imaging device according to any one of supplementary notes 1 to 8,
An imaging unit that performs the stereo imaging according to the control of the imaging control unit with respect to a housing, and a fixing unit that can be fixed so that the imaging unit can image the road surface;
A moving part capable of moving the housing in the direction;
Have
A moving object including an imaging device.

[Supplementary note 11] An information processing apparatus for evaluating road surface characteristics of a road as the object to be measured, based on the stereo-captured image output from the imaging device according to any one of supplementary notes 1 to 8. There,
A captured image acquisition unit that acquires the stereo captured image output from the imaging device according to any one of appendices 1 to 8;
A calculation unit for determining a pavement maintenance index for evaluating the road surface characteristics based on the stereo image acquired by the image acquisition unit;
An information processing apparatus comprising:

[Supplementary Note 12] The calculation unit includes:
Three-dimensional information is generated on the basis of the stereo image, the depth distance is determined in the width direction of the road on the basis of the generated three-dimensional information, and the rutting amount used for calculating the maintenance index of the pavement on the basis of the depth distance The information processing apparatus according to appendix 11, which calculates

[Supplementary Note 13] The calculation unit includes:
Three-dimensional information is generated based on the stereo image, the depth distance of the road surface on the road is determined based on the generated three-dimensional information over the surface direction of the road surface, and the maintenance management index of the pavement is calculated based on the depth distance. The information processing apparatus according to appendix 11, wherein a value indicating flatness used for calculation is calculated.

[Supplementary Note 14] The calculation unit includes:
The information processing apparatus according to appendix 11, wherein the image analysis is performed on the stereo image and the crack rate of the road surface on the road is calculated based on the result of the image analysis.

[Supplementary Note 15]
Three-dimensional information is generated based on the stereo image, the depth distance is determined in the width direction of the road based on the generated three-dimensional information, and a rutting amount is calculated based on the depth distance.
Obtaining the depth distance of the road surface on the road based on the three-dimensional information over the surface direction of the road surface, calculating a value indicating flatness based on the depth distance,
Perform image analysis on the stereo image, calculate the crack rate of the road surface on the road based on the result of the image analysis,
The information processing apparatus according to appendix 11, wherein a pavement maintenance index is obtained based on the calculated rutting amount, the flatness value, and the crack rate.

[Supplementary Note 16] Information processing for evaluating the road surface characteristics of the road as the object to be measured based on the stereo captured image output from the imaging device according to any one of Supplementary Notes 1 to 8 An information processing program to be executed by a device,
A captured image acquisition step of acquiring the stereo captured image output from the imaging device according to any one of appendices 1 to 8;
A calculation step of determining a pavement maintenance index for evaluating the road surface characteristics based on the stereo captured image acquired by the captured image acquisition step;
An information processing program for causing an information processing apparatus to execute

1 Vehicle 1a Case 1b Moving Part 2 Fixed Part 4 Road Surface 5 PC
6, 6L, 6R, 6Ln, 6Rn Stereo camera 6LL, 6LR, 6RL, 6RR Imaging lens 10, 10n Image processing system 10A Imaging device 50, 50n Information processing device 60C, 60L, 60R Stereo imaging range 60CL, 60CR, 60LL, 60LR , 60RL, 60RR Imaging range 100-1, 100-2 Imaging units 101-1, 101-2, 503 Imaging control units 102, 611, 5021 Speed acquisition device 103 Generation unit 500 Imaging image acquisition unit 503 Imaging control unit 510 Matching processing Unit 511 3D information generation unit 520 3D information acquisition unit 521 Condition characteristic value calculation unit 522 Report preparation unit 600L, 600R Imaging optical system 601L, 601R Imaging device 602L, 602R Drive unit

JP 07-318342 A

Claims (12)

Multiple imaging including distance information in the depth direction captured by overlapping at a predetermined overlapping rate with the width direction of the traveling surface of the moving object as an imaging range by multiple stereo imaging means configured by imaging elements provided with pixels of multiple lines An acquisition means for acquiring an image;
A plurality of depth maps are generated for the acquired plurality of captured images, a matching process is performed to match relative positions of the plurality of depth maps, and a plurality of depth maps aligned by the matching process are used as the traveling surface. Image processing means connected in the width direction of the image, and combined into a single depth map;
Measurement means for measuring the rutting amount of the traveling surface;
Equipped with
The matching process performed by the image processing unit combines the plurality of depth maps into one depth map in accordance with a position where both the depth distance of the overlapping region in the plurality of depth maps and the luminance of the pixels correspond. And
The measuring means measures the rutting amount of the traveling surface based on distance information in the width direction of the traveling surface obtained from the combined one depth map.
An information processing apparatus characterized by Control means for controlling the plurality of stereo imaging means to take images synchronously;
An information processing apparatus according to claim 1, characterized in that. The image processing means connects so that the distance information of the pixels of the overlapping area correspond to each other.
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The image processing unit connects the captured images so that the luminances of pixels in overlapping regions also correspond to each other.
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The measuring means measures the rutting amount excluding the end of the running surface,
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The measuring means measures the rutting amount by excluding foreign matters in the running surface.
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. The stereo imaging means includes an image pickup device of a simultaneous exposure batch readout method that reads out all pixels at the same timing.
The information processing apparatus according to any one of claims 1 to 6, characterized in that: A plurality of stereo imaging means;
An information processing apparatus according to any one of claims 1 to 7.
A fixing portion that can be fixed to the housing so that the plurality of stereo imaging means can image the traveling surface;
A movable unit capable of moving the housing;
A moving object comprising: A plurality of stereo imaging means configured to include an image sensor including pixels of a plurality of lines, and to overlap and image in the width direction of the traveling surface of the moving body;
Acquisition means for acquiring a plurality of captured images including distance information in the depth direction, which are captured by overlapping the width direction of the traveling surface as an imaging range at a predetermined overlap rate from the moving body, and the acquired plurality of captured images Generating a plurality of depth maps, performing a matching process for matching the relative positions of the plurality of depth maps, connecting the plurality of depth maps aligned by the matching process in the width direction of the running surface, Image processing means for combining with a depth map, and measuring means for measuring the amount of rutting on the running surface, and the matching processing performed by the image processing means is performed on the overlapping regions in the plurality of depth maps. Combining the plurality of depth maps into a single depth map in accordance with the position corresponding to both the depth distance and the luminance of the pixel ,
The measuring means is an information processing device that measures the rutting amount of the traveling surface based on distance information in the width direction of the traveling surface obtained from the combined one depth map;
An image processing system comprising: And
A fixing portion that can be fixed so that the stereo imaging means can capture the traveling surface with respect to the housing;
A moving unit for moving the housing along a running surface;
Comprising a moving body comprising
The image processing system according to claim 9. Multiple imaging including distance information in the depth direction captured by overlapping at a predetermined overlapping rate with the width direction of the traveling surface of the moving object as an imaging range by multiple stereo imaging means configured by imaging elements provided with pixels of multiple lines An acquisition step of acquiring an image;
A plurality of depth maps are generated for the acquired plurality of captured images, a matching process is performed to match relative positions of the plurality of depth maps, and a plurality of depth maps aligned by the matching process are used as the traveling surface. Image processing steps that are connected in the width direction and combined into a single depth map,
A measurement step of measuring a rut amount of the traveling surface;
Including
The matching process performed in the image processing step combines the plurality of depth maps into one depth map in accordance with a position where both the depth distance of the overlapping region in the plurality of depth maps and the luminance of the pixels correspond. And
The measuring step measures the rutting amount of the traveling surface based on the distance information in the width direction of the traveling surface obtained from the combined one depth map.
An information processing method characterized by the above. The image processing step includes
Determining a position to connect at a location where relative positions match in the plurality of depth maps;
Combining the plurality of depth maps into one depth map at the determined connecting positions;
Generating 3D point cloud information from the combined depth map;
The information processing method according to claim 11, further comprising: