JP2018036769A - Image processing apparatus, image processing method, and program for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to specify the position of a shadow caused by the sunlight in a photographed image.SOLUTION: An image processing apparatus 200 comprises: an image data acquisition part 201 that acquires image data of an image of an object photographed with a camera; a point group data acquisition part 202 that acquires point group data acquired with a laser scanner; a shadow area point group calculation part 204 that calculates, on the basis of the position of the sun, a shadow area point group obtained by projecting the point group data to the object from the sun; and a shadow area point group projection part 205 that projects the shadow area point group to the image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for specifying a position of a shadow reflected in an image.

  For example, there is a technique for inspecting roads for cracks, deterioration, and the like by capturing a road as a moving image or a still image and analyzing the captured image with software (see, for example, Patent Document 1).

JP 2011-179874 A

  By the way, in the above technique, there is a problem that the shadow of the building and the edge thereof (the boundary part between the shadow and the non-shadow) are obstructed in the image analysis due to sunlight. First, since the shadow portion has low brightness, the resolution and accuracy of image analysis are reduced. If it is known that this is a shadow portion, an analysis with accuracy can be performed by adjusting and analyzing an image suitable for the analysis of the shadow portion. However, it is difficult to determine with software whether or not this is a shadow. It is possible for a person to visually recognize whether or not it is a shadow. However, if a person visually recognizes the work, the amount of work increases and the meaning of using image analysis technology decreases.

  In addition, there is a problem of erroneously detecting the edge portion of the shadow as a crack formed on the road surface. If this problem is also confirmed by human eyes, erroneous detection can be avoided, but this makes the meaning of using image analysis technology lower. The purpose of image analysis by software processing is to reduce the work performed by humans as much as possible and improve the work efficiency. Therefore, it is desirable to minimize the work that humans see and judge.

  In such a background, an object of the present invention is to provide a technique for specifying the position of a shadow caused by sunlight in a captured image.

  The invention according to claim 1 is an image data acquisition unit that acquires image data of an image of an object photographed by a camera, a point cloud data acquisition unit that acquires point cloud data acquired by a laser scanner, and a sun position. A shadow region point group calculation unit that calculates a shadow region point group obtained by projecting the point group data from the sun onto the object, and a shadow region point group projection unit that projects the shadow region point group onto the image. An image processing apparatus comprising:

  The invention according to claim 2 is the invention according to claim 1, wherein the point cloud data acquired by the laser scanner is converted into a ground coordinate system and the point cloud data described in the ground coordinate system. Is further provided with a coordinate conversion unit that performs coordinate conversion into a camera coordinate system, and the calculation of the shadow area point group is performed in the ground coordinate system. Here, the coordinate conversion of the point cloud data acquired by the laser scanner to the ground coordinate system is performed when the coordinate system (scanner coordinate system) handled by the laser scanner is directly converted to the ground coordinate system, and from the scanner coordinate system. It includes both cases in which coordinate conversion to another coordinate system such as an IMU coordinate system is performed and then coordinate conversion to the ground coordinate system is performed.

  According to a third aspect of the present invention, in the first or second aspect of the present invention, in the image according to the first aspect, a shadow area expansion is performed to extend a shadow area around the shadow area point group based on a difference in luminance value of a pixel depending on a position. It comprises a part.

  The invention according to claim 4 is the invention according to any one of claims 1 to 3, further comprising a contrast adjusting unit that adjusts a contrast of the shadow region projected on the image in the image. And

  The invention according to claim 5 is the invention according to claim 3, further comprising a shadow boundary specifying unit that specifies a boundary of a shadow in the image based on the extended shadow region. .

  According to a sixth aspect of the present invention, in the fifth aspect of the present invention, the crack portion detection unit that detects a crack portion to be photographed from the image, and the position of the boundary portion in the image. And an erroneous detection determination unit that determines erroneous detection of a crack portion detected by the partial detection unit.

  The invention according to claim 7 is the invention according to claim 3, further comprising an overhead line detection unit that detects an overhead line based on the point cloud data, wherein the shadow region point group calculation unit is configured to detect the detected overhead line. The point cloud of the part used as the shadow of a line is calculated.

  The invention according to claim 8 is characterized in that the overhead line detection unit performs a process of predicting a position where the overhead lines continuously exist based on a point group of the overhead lines.

  The invention according to claim 9 is an image data acquisition step for acquiring image data of an image of an object photographed by a camera, a point cloud data acquisition step for acquiring point cloud data acquired by a laser scanner, and a sun position. A shadow area point group calculating step for calculating a shadow area point group obtained by projecting the point cloud data from the sun onto the object, and a shadow area point group projecting step for projecting the shadow area point group onto the image. It is an image processing method characterized by comprising.

  The invention according to claim 10 is a program to be read and executed by a computer, an image data acquisition unit for acquiring image data of an image of an object photographed by the camera by the computer, and a point cloud acquired by the laser scanner A point cloud data acquisition unit that acquires data, a shadow region point cloud calculation unit that calculates a shadow region point cloud obtained by projecting the point cloud data from the sun onto the object based on the position of the sun, and the shadow region point An image processing program that causes a group to function as a shadow area point group projection unit that projects a group onto the image.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which pinpoints the position of the shadow resulting from sunlight in the image | photographed image is obtained.

It is a conceptual diagram of embodiment. It is explanatory drawing of the principle of embodiment. It is a block diagram of an embodiment. It is a flowchart which shows an example of the procedure of a process. It is explanatory drawing explaining a shadow area | region point group. It is an image figure of the camera image which projected the shadow area | region point group.

(Overall overview)
FIG. 1 shows a vehicle 100. The vehicle 100 includes a laser scanner 101, a camera 102, an IMU (Inertial Measurement Unit) 103, a GNSS device 104, and a storage device 105. The laser scanner 101 performs laser scanning of the surrounding environment and acquires point cloud data. As the laser scanner 101, the techniques described in JP 2010-151682 A, JP 2008-268004 A, and US Pat. No. 8,767,190 can be used.

  The camera 102 captures images at specific time intervals and obtains image data of the captured images. A camera 102 that captures moving images can also be used. In this case, a frame image is used as a captured image. The IMU 103 is equipped with a three-axis acceleration, three-axis angular velocity, three-axis geomagnetic nine-axis motion sensor, and measures changes in posture and inclination. The GNSS device 104 receives a navigation signal from a navigation satellite such as a GPS satellite and measures a position based on the received navigation signal. The GNSS device 104 has a clock function that outputs highly accurate time information based on a navigation signal from a navigation satellite.

  The storage device 105 is a data storage device configured by a hard disk device or a semiconductor memory. The storage device 105 receives the point cloud data acquired by the laser scanner 101, the image data of the image captured by the camera 102, the attitude and tilt change data measured by the IMU 103, and the position and time data measured by the GNSS device 104. Store in the associated state. The external orientation elements (posture and position) of the laser scanner 101, the camera 102, the IMU 103, and the GNSS device 104 with respect to the vehicle 100 are measured in advance and are known.

(Description of principle)
The system of FIG. 1 is used to detect cracks in a road paved with concrete by image analysis. In this example, a technique for reducing the problem of erroneous detection of cracks due to the influence of shadows not exposed to sunlight will be described.

  The shadow in this specification refers to a portion that is not directly exposed to sunlight, which is formed when sunlight strikes a building or the like. As shown in FIG. 2, when the shadow of a building is reflected on the road surface, a linear boundary line is generated. This boundary line is erroneously detected as a crack on the road surface. In this example, the three-dimensional position of the building is grasped by laser scanning, and a shadowed area that is not exposed to sunlight is calculated from the three-dimensional information (point cloud data). Since the position of the sun on the celestial sphere at a specific time can be precisely obtained from astronomical data and calculation formulas, the position of the shadow reflected on the road surface can be obtained by geometric calculation.

  Then, by projecting the position of the shadow on the road surface in the image taken by the camera 102, image data that shows where the shadow is reflected in the image is created, and it is possible to determine erroneous detection of cracks. In addition, since a shadow portion in the image can be specified in advance, the detection accuracy of a crack in the shadow portion can be increased by adjusting the contrast of the portion.

(Configuration of image processing unit)
FIG. 3 shows a block diagram of the image processing apparatus. FIG. 3 shows an image processing apparatus 200. The image processing apparatus 200 has a basic function as a computer. The image processing apparatus 200 is an apparatus for detecting a crack on the road surface by image analysis. In particular, the shadow portion in the image is specified using the principle shown in FIG. 2, and further, the accuracy of detecting the crack on the road surface due to the presence of the shadow is reduced. The process which suppresses is performed.

  The image processing apparatus 200 includes an image data acquisition unit 201, a point group data acquisition unit 202, a coordinate conversion unit 203, a shadow region point group calculation unit 204, a shadow region point group projection unit 205, a shadow region expansion unit 206, and a shadow region specification unit. 207, a contrast adjusting unit 208, a shadow boundary line specifying unit 209, a cracked part detecting unit 210, an erroneous detection determining unit 211, an overhead line detecting unit 212, a storage unit 213, and a vehicle position specifying unit 214.

  The functional unit included in the image processing apparatus 200 described above may be configured as software, or may be configured as a dedicated arithmetic circuit. Moreover, the function part comprised by software and the function part comprised by the dedicated arithmetic circuit may be mixed. For example, each functional unit shown in the figure is configured by an electronic circuit such as a PLD (Programmable Logic Device) such as a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

  Whether each functional unit is configured by dedicated hardware or by software by executing a program in the CPU is determined in consideration of required calculation speed, cost, power consumption, and the like. For example, if a specific functional unit is configured with FPGA, it is advantageous in terms of processing speed but is expensive. On the other hand, a configuration in which a specific functional unit is realized by executing a program by the CPU can save hardware resources, and is advantageous in terms of cost. However, when the functional unit is realized by the CPU, the processing speed is inferior to that of dedicated hardware. Further, when the functional unit is realized by the CPU, it may not be able to cope with complicated calculations. It should be noted that the configuration of the functional unit with dedicated hardware and the configuration with software are equivalent from the viewpoint of realizing a specific function, although there are the differences described above.

  In this example, the image processing apparatus 200 is configured by a computer such as a workstation that can process a large amount of data at high speed. Of course, if there is no problem in processing capability, the image processing apparatus 200 can be configured using a commercially available personal computer. Further, a configuration in which the function of the image processing apparatus 200 is distributed and processed by a plurality of computers is also possible.

  The image data acquisition unit 201 acquires image data of an image captured by the camera 102. The image data is frame image data constituting a still image or a moving image taken at a specific time interval. The point cloud data acquisition unit 202 acquires the point cloud data acquired by the laser scanner 101. Point cloud data is data that captures a scan target as a collection of points from which three-dimensional coordinates have been acquired.

  The coordinate conversion unit 203 relates to a coordinate system that describes point cloud data, coordinate conversion from the scanner coordinate system to the IMU coordinate system, coordinate conversion from the IMU coordinate system to the ground coordinate system, and conversion from the ground coordinate system to the IMU coordinate system. Coordinate transformation, coordinate transformation from IMU coordinate system to camera coordinate system is performed. The scanner coordinate system is a coordinate system with the laser scanner 101 as the origin. The IMU coordinate system is a coordinate system having the IMU 103 as an origin. The ground coordinate system is a coordinate system fixed on the ground. The camera coordinate system is a coordinate system with the camera 102 as the origin.

  In the scanner coordinate system, the IMU coordinate system, and the camera coordinate system, as the vehicle 100 moves, the position of the origin and the direction of the coordinate axis change every moment. Of course, if the vehicle 100 does not move, there is no movement of the origin of coordinates, and if there is no change in the direction of the vehicle 100, there is no change in the direction of the coordinate axes of the coordinate system.

  Coordinate conversion from the scanner coordinate system to the IMU coordinate system is performed based on external orientation elements of the laser scanner 101 and the IMU 103 with respect to the vehicle 100. More specifically, the coordinate conversion from the scanner coordinate system to the IMU coordinate system is performed as follows. First, consider a specific time t. The relative relationship between the scanner coordinate system and the IMU coordinate system at time t can be determined from the external orientation elements of the laser scanner 101 and the IMU 103 with respect to the vehicle 100. That is, the scanner coordinate system can be converted to the IMU coordinate system by translating and rotating the scanner coordinate system at time t. The conversion coefficient at this time is obtained from external orientation elements of the laser scanner 101 and the IMU 103 with respect to the vehicle 100. Therefore, the coordinate value in the IMU coordinate system of the specific point A in the scanner coordinate system at time t can be calculated. By performing the same process at an arbitrary time, coordinate conversion from the scanner coordinate system to the IMU coordinate system at the time is performed.

  Conversion from the IMU coordinate system to the ground coordinate system is performed using information on the vehicle trajectory. Information on the vehicle trajectory is obtained from the IMU 103 and the GNSS device 104. More specifically, coordinate conversion from the IMU coordinate system to the ground coordinate system is performed as follows. First, consider a specific time t. The relative relationship between the IMU coordinate system and the ground coordinate system at time t is obtained from the external orientation elements of the IMU 103 with respect to the vehicle 100 and the position and orientation of the vehicle 100 at time t. Information on the position of the vehicle 100 is obtained from the GNSS device 104, and information on the direction of the vehicle 100 is obtained from the IMU 103. In this case, the IMU coordinate system can be converted to the ground coordinate system by translating and rotating the IMU coordinate system at time t.

The conversion coefficient at this time is obtained from the external orientation element of the IMU 103 with respect to the vehicle 100 and the position and orientation of the vehicle 100 at time t. Therefore, the coordinate value in the ground coordinate system of the specific point A in the IMU coordinate system at time t can be calculated. Then, by performing the above processing at times t ₁ , t ₂ , t ₃ ..., Points acquired at each time at times t ₁ , t ₂ , t _3. Data can be converted to the ground coordinate system.

  Note that the time interval in the coordinate conversion of the scanner coordinate system → IMU coordinate system → ground coordinate system is determined by the scanning speed of the laser scanner 101. The scanning speed of the laser scanner 101 is, for example, about 1 kHz to 10 kHz. The narrower (shorter) the time interval, the higher the accuracy, but the greater the amount of computation.

Conversion of the point cloud data from the ground coordinate system to the IMU coordinate system is performed as follows. This conversion is performed at each time marked as times T ₁ , T ₂ , T ₃ . The time Tn at this time is the shooting time of the camera 102. The shooting interval of the camera 102 is, for example, about 0.1 seconds to 0.5 seconds. Of course, the image may be taken at a time interval shorter than this, but the burden required for calculation increases.

First, the relationship between the ground coordinate system and the IMU coordinate system at time T ₁ is obtained from the external orientation element of the IMU 103 and the position and orientation of the vehicle at time T ₁ (this data is obtained from the IMU 103). Thereby, the coordinate conversion of the point cloud data at the time T ₁ from the ground coordinate system to the IMU coordinate system becomes possible, and the point cloud data described in the IMU coordinate system at the time T ₁ is obtained. By performing this processing also at other times, point cloud data in the IMU coordinate system at the photographing time Tn can be obtained.

  The coordinate conversion of the point cloud data from the IMU coordinate system to the camera coordinate system is performed based on the external orientation elements of the camera 102 and the IMU 102 according to the same principle as the coordinate conversion from the scanner coordinate system to the IMU coordinate system. Since the IMU coordinate system and the camera coordinate system move together with the vehicle 100, coordinate conversion from the IMU coordinate system to the camera coordinate system is performed by designating the time Tn and obtained at the designated time Tn.

  Instead of converting the scanner coordinate system → the IMU coordinate system → the ground coordinate system, it is possible to convert the scanner coordinate system → the ground coordinate system without going through the IMU coordinate system. In either case, the result is a conversion from the scanner coordinate system to the ground coordinate system. Further, instead of converting from the ground coordinate system → IMU coordinate system → camera coordinate system, it is possible to convert from the ground coordinate system → camera coordinate system without going through the IMU coordinate system. In this case, the result is a transformation from the ground coordinate system to the camera coordinate system.

  Since the scanner coordinate system changes every moment in the ground coordinate system according to the movement of the vehicle, by converting each point to the ground coordinate system, it can be made into a single data (point cloud), and the ground coordinate system It is possible to project a building point group (a shadow region point group constituting the shadow of the building) from the sun direction to the road surface point group only after the point group becomes. The shadow area point group in the ground coordinate system obtained in this way is converted into the IMU coordinate system at the time Tn when the image was taken, and it is converted into the camera coordinate system and projected onto the captured image of the camera. In this method of once converting to the ground coordinate system, the point cloud data can be easily handled even if the scan timing of the laser scanner 101 and the shooting time of the camera 102 are not synchronized. Therefore, the degree of freedom in selecting the models of the laser scanner 101 and the camera 102 and setting the operation timing is high, and the system can be configured at low cost.

  Hereinafter, an example of processing performed by the coordinate conversion unit 203 will be described. First, a three-dimensional coordinate system (first coordinate system: scanner coordinate system) having the laser scanner 101 as the origin is considered as the coordinate system of the point cloud data. On the other hand, a three-dimensional coordinate system (second coordinate system: camera coordinate system) with the camera 102 as the origin is considered. Here, it is assumed that the external orientation elements (position and orientation in the coordinate system handled by the IMU 103) of the laser scanner 101 and the external orientation elements (position and orientation in the coordinate system handled by the IMU 103) of the camera 102 are known. A three-dimensional coordinate system handled by the IMU 103 is a third coordinate system (IMU coordinate system).

  Since the external orientation element of the laser scanner 101 is known, the first coordinate system can be converted into the third coordinate system (IMU coordinate system) by rotating and translating the first coordinate system. Further, if the position and orientation of the IMU 103 at the time of acquisition of each point acquired by the laser scanner are known in the fourth coordinate system (ground coordinate system), the point cloud data can be described on the fourth coordinate system. . On the other hand, since the external orientation element of the camera 101 is known, the second coordinate system can be converted into the third coordinate system by rotating and translating the second coordinate system. Further, if the position and orientation of the IMU 103 at the photographing time are known in the fourth coordinate system, the position and orientation of the camera 102 at the time can be converted to the fourth coordinate system (ground coordinate system). As a result, it is possible to describe, for example, the direction of the optical axis of the camera 102 and the angle of view (imaging field of view) on the fourth coordinate system (ground coordinate system). Of course, an operation of converting a point group in the ground coordinate system to the second coordinate system (camera coordinate system) by mathematical operation is possible.

  Based on the position of the sun, the shadow area point group calculation unit 204 calculates, as the shadow area point group, a point group of a shadow area that does not receive sunlight due to the presence of the point cloud data. This process is performed on the ground coordinate system. By performing this processing on the ground coordinate system, the point cloud data can be handled as a set of data, the processing time is shortened, and the burden of hardware for processing is reduced. As the sun moves, so does the shadow. Therefore, the calculation of the shadow area point group is repeated at specific time intervals. Specifically, the shadow area point group is calculated in synchronization with the shooting timing of the camera 102 (for example, every 0.1 second). In order to reduce the burden of calculation, a mode is also possible in which the shadow area point group is calculated at a rate of once per a plurality of shooting timings of the camera 102. For example, when the shooting interval is 0.1 seconds, it is possible to calculate the shadow area point group every 0.5 seconds.

  Hereinafter, an example of calculating the shadow area point group will be described. For example, when the sun direction is seen from a certain viewpoint A on the road surface, a case is considered in which there is a building in that direction and the sun cannot be directly seen. In this case, there is a point B constituting the building on the line connecting the viewpoint A and the sun, and this point B is acquired as one of the point groups when laser scanning is performed from the viewpoint. A point obtained by projecting this point B onto the road surface with the sun as the viewpoint (origin) is one of the shadow area point groups. This projection point coincides with the viewpoint A on the previous road surface. FIG. 5 shows an example of the relationship between the point group obtained by laser scanning and the shadow area point group.

  In the process of calculating the shadow area point cloud, first, the sun at infinity is used as a light source, and the point cloud acquired by the point cloud data acquisition unit 202 (the point cloud measured by the laser scanner 101) is projected on the ground (road surface). By this projection, a group of projection points on the ground corresponding to each point group measured by the laser scanner 101 is obtained as a shadow projection point group. The shadow projection point group is a point group composed of point-like shadows.

  For example, data obtained from the NASA's Jet Propulsion Laboratory site is used as the data relating to the position of the sun and its calculation program. Of course, the position of the sun may be specified using a unique measurement value or calculation program. For example, data related to the position of the sun is described in science chronology, and many calculation methods and calculation programs based on the data are disclosed. Based on this information, if the position and time of the viewpoint are known, the position of the sun on the celestial sphere can be accurately obtained by calculation.

  Hereinafter, an example of processing for calculating a shadow area point group will be described. First, point cloud data in the ground coordinate system is generated by converting each point acquired by the laser scanner 101 from the scanner coordinate system (the first coordinate system) to the ground coordinate system based on the acquisition time of each point. Next, the position (the position of the camera 102) in the fourth coordinate system (ground coordinate system) of the origin of the second coordinate system (camera coordinate system) at the acquisition time of the image acquired by the camera 102 and the time are the sun position. To the data table or program for obtaining the position of the sun on the celestial sphere at the time, that is, the direction in which sunlight falls in the fourth coordinate system (ground coordinate system).

  Next, in the point group data in the fourth coordinate system (ground coordinate system), a point-like portion where a shadow is formed without being exposed to sunlight on the road surface (the ground surface) is calculated. This process is executed by performing a process of projecting the point cloud acquired by the laser scanner 101 on the ground with the sun as the projection origin. For example, if there is a point group between the optical axis of sunlight and a point on the ground surface, the point on the ground surface becomes a shadow portion. A group of points that become shadows becomes a shadow area point group. By performing this process on all the point clouds obtained by the point cloud data acquisition unit 202, it is possible to determine an area that is a shadow on the ground surface. FIG. 5 shows an example of a point cloud acquired by a laser scanner by a laser scanner and a shadow area point cloud obtained by projecting this point cloud onto a road surface.

  The shadow area point group projection unit 205 projects the shadow area point group calculated by the shadow area point group calculation unit 204 on the image captured by the camera 102 using the result of the coordinate conversion performed by the coordinate conversion unit 203. This process makes it possible to visually grasp how the point group measured by the laser scanner 101 and the shadow area point group obtained in the ground coordinate system look in the image captured by the camera 102.

  Hereinafter, an example of processing performed by the shadow area point group projection unit 205 will be described. First, the direction of the optical axis and the angle of view (imaging field of view) of the camera 102 are described on the fourth coordinate system (ground coordinate system) from the result of the coordinate conversion between the first to fourth coordinate systems. Is possible. On the other hand, the shadow area point group described above can be described on the fourth coordinate system (ground coordinate system). Therefore, the position of the shadow area point group in the shooting range (captured screen) of the camera 102 can be specified. That is, it is possible to project a shadow area point group on the shooting range (captured screen) of the camera 102 and display the shadow area point group in the shooting range.

  By projecting the shadow area point cloud in the image photographed by the camera 102, image information regarding which part becomes a shadow based on the point cloud data is displayed in the image photographed by the camera 102. FIG. 6 shows an image diagram of a camera image in which a shadow area point group is projected. FIG. 6 shows a state in which a shadow area point group constituting a shadowed part of a building is projected in an image obtained by photographing a road surface. In the case of FIG. 6, the sun is in the upper left direction of the screen, and a state in which a shadow area point group is formed in an area corresponding to a shadow generated when the sunlight is blocked by a building is shown. The shadow area point cloud is based on the point cloud data of the building obtained by laser scanning and the calculated sun position, and it does not matter how the shadow is actually reflected in the image. Therefore, the position of the shadow in the shooting screen (camera image) is specified by the shadow area point group regardless of how the shadow of the building is reflected in the screen.

  The shadow area expanding unit 206 extends a shadow area around a shadow area point group in an image (camera image) captured by the camera 102 based on the luminance value of the pixels constituting the image. In general, the point density of a point group is coarser than that of an image pixel. Since the shadow area point group is a point group composed of point-like shadows, the density is coarse compared to the pixels of the image. Therefore, in order to determine the boundary portion between the shadow and the non-shadow portion with high accuracy, it is necessary to extend the shadow region to an adjacent region of the shadow region point group. This process is performed by the shadow area expansion unit 206.

  Hereinafter, an example of processing performed by the shadow area expansion unit 206 will be described. First, a point at the edge of the shadow area point group (this point is the point of interest) is extracted. Next, the luminance value of the pixels constituting the image is examined from this point toward the periphery (along the radial direction). The range to be examined is 360 ° around and is about twice the distance to the adjacent point of interest.

  Here, the gradient of the change in the luminance value of a plurality of pixels located in a linear array is examined. The position where the gradient exceeds the threshold value is acquired as a shadow boundary point. For example, when the gradient of the luminance values of three pixels arranged in a straight line is examined, the central pixel portion of the group of three pixels having the maximum gradient is acquired as a boundary point. This boundary point is acquired for one or more points of interest, and for all the points of interest as targets. Then, the outline of the shadow region is determined by connecting the obtained plurality of boundary points. The inside of the outline of the shadow area is painted as a shadow.

  The shadow specifying unit 207 specifies a shadow part (a part of the ground surface (road surface) that is not exposed to sunlight) defined by the processing in the shadow area extending unit 206. The contrast adjustment unit 208 adjusts the contrast of the shadow portion on the road surface. Specifically, on the premise that the image is a shadow and a dark image, the contrast is adjusted so that the display is enhanced. By performing contrast processing on the assumption that the image is a shadow portion and a dark image, the detection accuracy of a cracked portion of the road surface in the shadow portion can be increased.

  The shadow boundary line specifying unit 209 specifies the edge of the shadow part (boundary between the shadow and the non-shadow part) defined by the process in the shadow area extending unit 206. The edge portion of the shadow is linear, and it may be erroneously detected that this linear portion is a cracked portion of the road surface. By specifying the edge portion of the linear shadow that may be erroneously detected in advance, the reliability of the finally obtained data can be improved.

  The crack portion detection unit 210 detects a crack portion on the road surface from the image captured by the camera 102 by image analysis. This process is performed using a normal image recognition technique. As this technique, the technique described in Patent Document 1 described in the “Prior Art” column can be used.

  The error detection determination unit 211 determines the correctness of the crack portion detected by the crack portion detection unit 210 based on the shadow boundary line (shadow edge) obtained by the shadow boundary line specifying unit 209. In this process, the position of the shadow boundary line on the shooting screen is compared with the position of the cracked portion detected by the cracked portion detection unit 210 on the shooting screen, and the two can be regarded as matching or matching. When it is recognized that there is a match in the error range, it is determined that the detection by the cracked portion detection unit 210 was an error. By this processing, the reliability of the inspection data can be improved.

  The overhead line detection unit 212 detects an overhead line installed in the sky. Examples of overhead lines include electric wires, telephone lines, optical fiber cables, cable TV signal lines, utility pole branch lines (stay lines), and the like. Depending on the scan density of the laser scanner 101, some of the overhead lines may be acquired as point cloud data. Since the overhead line is in the sky, there is no point group such as another building in the vicinity, and the line is isolated and extends linearly. Therefore, when two or more point groups separated in an isolated space are detected, they are connected and specified as an overhead line. It is determined using a threshold value whether or not it is present in isolation or away from a building or the like. In addition, image data obtained by capturing an image of the sky can also be used to identify point cloud data that is assumed to be an overhead line obtained from the laser scan data described above. In this case, however, a part of the overhead line must be detected.

  The overhead line detection unit 212 obtains the position data of the overhead line by calculation on an appropriate coordinate system such as ground coordinates. For example, when an isolated point group of two points is obtained, an overhead line connecting the two points is set, an equation of the straight line or a curve is obtained, and the coordinates are obtained. That is, the overhead line is linear, and the scan light is not applied, and the point cloud may be missed. Alternatively, there are cases where point cloud data can be obtained only for a part of the overhead line. In this case, paying attention to the fact that the overhead line extends linearly, the point cloud of the missing part is predicted. For example, by preparing multiple straight line equations or curve equations that draw loose curves, and substituting the partially obtained overhead line point group into them, the line equation to be fitted to the obtained point group can be found. The position of the overhead line is specified by the equation of the line. The data of the position of the overhead line is used in the calculation performed by the shadow area point group calculation unit 204, and a shadow area point group corresponding to the overhead line is obtained. That is, a shadow area point group corresponding to a shadow caused by the presence of an overhead line is obtained.

  The shadow of the overhead line is linear and is likely to be detected as a crack on the road surface when projected onto the road surface (it is often difficult to identify in image analysis processing). By obtaining the shadow area point group caused by the overhead line, it is possible to reduce false detection caused by the overhead line, and to improve the reliability of the data related to the cracks on the road surface obtained.

  The storage unit 213 stores data handled by the image processing apparatus 200 and program data for operating the image processing apparatus 200. In addition, basic data necessary for calculation is stored in the storage unit 213.

  The vehicle position / orientation calculation unit 214 calculates the position and orientation of the vehicle 100 at the designated time t. Since the external orientation elements (position and orientation) of the laser scanner 101 and the camera 102 with respect to the vehicle 100 are known, if the position and orientation of the vehicle at the time t are obtained, the positions and orientations of the laser scanner 101 and the camera 102 at the time t are obtained. It is also obtained.

  The trajectory of the movement of the vehicle 100 is recorded momentarily by GNSS data and IMU data. From this data, the position and orientation of the vehicle 100 at the designated time t can be obtained by calculation. This process is performed by the vehicle position specifying unit 214. By specifying the position of the vehicle 100 at time t, the position of the sun at time t, the position of the origin of the scanner coordinate system, and the position of the origin of the camera coordinate system are determined. In addition, the orientation of each coordinate system is determined by obtaining the posture.

(Example of operation)
FIG. 4 shows an example of a processing procedure executed by the image processing apparatus 200. The program for executing the processing of FIG. 4 is stored in the storage unit 213 or an appropriate storage medium, and is read from and executed. The program can also be acquired from a server or a storage device at a remote location via an appropriate line.

  In the process of FIG. 4, the processes of steps S <b> 101 to S <b> 102 are performed on the vehicle 100 side, and the processes of steps S <b> 103 to S <b> 112 are performed on the image processing apparatus 200. First, the road surface is photographed at a specific time interval (for example, every 0.1 second) using the camera 102 while the vehicle is running. At this time, the laser scanning by the laser scanner 101 is performed in synchronization with the photographing of the camera 102 to obtain point cloud data of the surrounding environment. In laser scanning, a scan range is set so that a building that may form a shadow on a road surface and an overhead overhead line are scanned. The image data and the point cloud data, the time when these data were obtained, and the attitude and position of the vehicle at that time (the attitude and position of the IMU 103) are associated and stored in the storage device 105 (steps S101 and S102).

When a series of data is obtained, the data is transferred from the storage device 105 to the image processing device 200, and the processing from step S103 onward is executed. First, to obtain the position and orientation of the laser scanner 101 and the camera 102 at time _{t 1} (step S103). Here, the position and orientation data is calculated by the vehicle position and orientation calculation unit 214. This data is used when converting the scanner coordinate system and the camera coordinate system to the IMU coordinate system. Then, to get the position of the sun at the time _{t 1} (step S104), and calculates the shadow area point group at time _{t 1} (step S105). At this time, a shadow area point group is also calculated for the shadow of the overhead line.

After obtaining the shadow region point group is projected on the camera image captured it at time t ₁ (step S106). Next, extend the shadow area point group on the camera image at time t ₁ (step S107), and determine the area of the shadow in the relevant camera image (step S108).

  Next, adjustment for enhancing the contrast of the image portion determined as the shadow region is performed (step S109). By adjusting the contrast, the detection accuracy of cracks from a portion that becomes a shadow and becomes a dark image increases. Thereafter, image analysis processing for detecting a cracked portion of the road surface is performed (step S110), and the result is compared with the information on the shadow area obtained in step S108, and correct / incorrect determination is performed (step S111). By performing correct / incorrect determination, the inconvenience of identifying the edge of the shadow as a crack is avoided.

The above processing is repeated at time t ₂ , time t ₃ , time t ₄ ... And performed for all target camera images (step S 112).

  It is also possible to perform the processing of FIG. 4 in real time. In this case, the processing of FIG. 4 is performed while the image processing apparatus 200 is placed on the vehicle 100 and the vehicle is running.

(Superiority)
In the technology described above, the area of the road surface where the sunlight does not become a shadow of a building or the like is calculated using the point cloud data and the position of the sun. In image analysis, even if it is difficult to determine whether it is a shadow edge or a crack on the road surface, if the shadow part is known in advance, the problem is that the edge part of the shadow is recognized as a crack part by the correctness determination process. Is avoided. In addition, the shadow portion is dark and the crack detection accuracy decreases. However, if the shadow portion is obtained in advance from the point cloud data, the crack detection accuracy can be increased by adjusting the contrast of the portion. .

  In addition, the shadows of overhead lines such as electric wires are reflected on the road surface, which causes a problem of erroneously detecting the shadows of overhead lines as cracks, but if the information about the shadows of overhead lines is obtained in advance from point cloud data By eliminating the data of that part, the problem of identifying the shadow of the overhead line as a crack can be avoided.

(Other)
The present invention is not limited to the detection of cracks on the road surface, but can also be used for detection of cracks on slopes and cliffs hardened with concrete or mortar, detection of cracks on bridge piers, and the like.

Claims (10)

An image data acquisition unit for acquiring image data of an image of an object photographed by the camera;
A point cloud data acquisition unit for acquiring the point cloud data acquired by the laser scanner;
A shadow area point cloud calculation unit for calculating a shadow area point cloud obtained by projecting the point cloud data from the sun onto the object based on the position of the sun;
An image processing apparatus comprising: a shadow region point group projection unit that projects the shadow region point group onto the image. A coordinate conversion unit for converting the point cloud data acquired by the laser scanner into a ground coordinate system, and converting the point cloud data described in the ground coordinate system into a camera coordinate system;
The image processing apparatus according to claim 1, wherein the calculation of the shadow area point group is performed in the ground coordinate system.   The image processing apparatus according to claim 1, further comprising a shadow area extending unit that extends a shadow area around the shadow area point group based on a difference in luminance value of a pixel depending on a position in the image.   The image processing apparatus according to claim 1, further comprising a contrast adjustment unit that adjusts a contrast of the shadow area projected on the image in the image.   The image processing apparatus according to claim 3, further comprising a shadow boundary line specifying unit that specifies a boundary line of a shadow in the image based on the extended shadow region. A cracked part detecting unit for detecting a cracked part to be photographed from the image;
The image processing apparatus according to claim 5, further comprising: an erroneous detection determination unit that determines erroneous detection of a crack portion detected by the crack portion detection unit based on a position of the boundary portion in the image. An overhead line detection unit that detects an overhead line based on the point cloud data,
The image processing apparatus according to claim 3, wherein the shadow region point group calculation unit calculates a point group of a portion that is a shadow of the detected overhead line.   The image processing apparatus according to claim 7, wherein the overhead line detection unit performs processing for predicting a position where the overhead lines continuously exist based on a point cloud of the overhead lines. An image data acquisition step of acquiring image data of an image of the object photographed by the camera;
A point cloud data acquisition step for acquiring the point cloud data acquired by the laser scanner;
A shadow area point cloud calculating step for calculating a shadow area point cloud obtained by projecting the point cloud data from the sun onto the object based on the position of the sun;
An image processing method comprising: a shadow region point group projecting step of projecting the shadow region point group onto the image. A program that is read and executed by a computer,
An image data acquisition unit for acquiring image data of an image of an object captured by a camera of a computer;
A point cloud data acquisition unit for acquiring the point cloud data acquired by the laser scanner;
A shadow area point cloud calculation unit that calculates a shadow area point cloud obtained by projecting the point cloud data from the sun onto the object based on the position of the sun;
An image processing program that functions as a shadow region point group projection unit that projects the shadow region point group onto the image.

Images