JP6060682B2 - Road surface image generation system, shadow removal apparatus, method and program - Google Patents

Description

  The present invention relates to a road surface image generation system that generates a road surface image in which shadows caused by solar radiation are eliminated, and a shadow removal apparatus, method, and program that remove shadows caused by solar radiation from a road surface image.

  Real images of the ground, especially road surfaces, are required for car navigation systems and various maps. Such a road surface image is obtained by mounting a video camera on a vehicle, traveling while photographing a road surface such as a road, and synthesizing the obtained images in the road direction. In order to enable superimposition with various geospatial information, it is provided as an ortho image obtained by orthogonal projection transformation so that the image is viewed from above. In addition to road vehicles, aerial photography using a light or unmanned airplane may be used.

  Japanese Patent Laid-Open No. 2004-133867 video-shoots a range (for example, a range of 50 cm or less from the road surface) specified by the measurement value of the laser scanner by three-dimensionally measuring the target with a laser scanner while shooting the target including the road surface with a video camera. A technique for generating a road surface image in which a white line, a road boundary, and the like can be clearly recognized by synthesizing and displaying captured images of a camera so that a road is not blocked by a shield such as a tree or a tunnel is described.

  In Patent Document 2, a GPS receiver and an IMU (inertial unit) are mounted on a vehicle to measure the position of the vehicle, and at the same time, while photographing a road surface and its surroundings with a digital camera, three-dimensional measurement is performed with a laser scanner. A technique for forming a 3D model from point cloud data obtained by a laser scanner is described with reference to a feature and vehicle position data included in an image.

No. 2010/024212 Special table 2012-511697 gazette

  In the method described in Patent Document 1, when a shadow of solar radiation is included in a video image taken by a video camera, a road surface orthoimage from which the shadow is removed cannot be generated.

  In the technique described in Patent Document 2, although the road surface is photographed with a camera, the target shape is mainly determined by a three-dimensional measurement value of a laser scanner, and the camera image is used in a complementary manner. I can't make use of it.

  It is an object of the present invention to provide a road surface image generation system, a shadow removal device, a method, and a program that can remove a shadow caused by solar radiation and generate a road surface image that conforms to a real image.

  A road surface image generation system according to the present invention includes an imaging device that images a road surface in a shaded state due to solar radiation, and a road surface ortho image generation unit that generates road surface ortho image data from a plurality of image data obtained by the imaging device. A laser scanner that scans the same road surface with laser light and measures the reflection intensity of the reflection point; and the road surface orthoimage data from the reflection intensity data of the reflection point obtained by the laser scanner. A reference image generating means for generating a reference image data for comparison with a prescribed reflection intensity range, a shadow part recognizing means for referring to the reference image data and recognizing a shadow area of the road surface ortho image data, and A shadow that is corrected by the color information of the peripheral portion of the shadow area of the road surface orthoimage data for each shadow area recognized by the shadow part recognition means. Characterized by comprising the divided correction means.

  A shadow removal apparatus according to the present invention is a shadow removal apparatus that removes the shadow from image data obtained by photographing a road surface in a shaded state due to solar radiation, and obtains road surface ortho image data from the image data obtained by photographing the road surface. Reference image data for comparing road surface reflection intensity image data with the road surface ortho image data based on the road surface ortho image generation means and the reflection intensity data measured by scanning the same road surface with laser light with a laser scanner As a reference image generating means for generating a predetermined reflection intensity range, a shadow part recognizing means for recognizing a shadow area of the road surface ortho image data with reference to the reference image data, and the shadow part recognizing means. Shadow portion correction means for correcting each shadow region with the color information of the peripheral portion of the shadow region of the road surface orthoimage data is provided.

  A shadow removal method according to the present invention is a shadow removal method for removing a shadow from image data obtained by photographing a road surface in the state of being shaded by solar radiation, and recording road surface video data obtained by photographing the road surface with a video camera. From the information processing apparatus, the road surface ortho image generation means of the information processing apparatus generates road surface ortho image data from the road surface image data, and the information processing apparatus reference The image generation means uses the reflection intensity data measured by scanning the same road surface with a laser scanner to scan the road surface reflection intensity image data as reference image data for comparison with the road surface ortho image data. The reference image generation step for generating in the intensity range and the shadow part recognition means of the information processing apparatus refer to the reference image data. A shadow part recognition step for recognizing a shadow area of the road surface ortho image data, and a shadow part correction unit of the information processing apparatus, for each shadow area recognized by the shadow part recognition unit, of the road surface ortho image data, And a shadow part correction step for correcting with the color information of the peripheral part of the shadow region.

  In order to remove the shadow from image data obtained by photographing a road surface in the state of being shaded by solar radiation, the shadow removal program according to the present invention is configured to change the information processing device from the image data obtained by photographing the road surface to road surface ortho-image data. The road surface orthoimage generating means for generating the same and the reference image for comparing the road surface reflection intensity image data with the road surface orthoimage data from the reflection intensity data measured by scanning a laser beam with the laser scanner on the same road surface Reference image generation means for generating data within a prescribed reflection intensity range, shadow part recognition means for recognizing the shadow area of the road surface ortho image data with reference to the reference image data, and the shadow part recognition means. Each of the shadow areas is functioned as a shadow part correction unit that corrects the color information of the peripheral part of the shadow area of the road surface ortho image data. It is those of.

  According to the present invention, since a shadow portion can be converted into a shade without a shadow from a road surface image photographed by an imaging device, it is not necessary to intentionally avoid the reflection of a shadow when photographing a road surface.

1 shows a schematic block diagram of an embodiment of the present invention. The positional relationship on the road surface of the imaging | photography range by a video camera and the laser scanning locus | trajectory by a laser scanner is shown. It is a TIN (Triangulated Irregular Network) model example explaining road surface area extraction. It is explanatory drawing of a normal vector. The flowchart of normal vector calculation is shown. The flowchart of road surface determination is shown. The operation | movement flowchart of a shadow part recognition process and a shadow part correction process is shown. It is explanatory drawing of the smoothing process of the non-shadow area | region adjacent to a shadow area | region.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of an embodiment of the present invention.

  A video camera 10, a one-dimensional scanning capability laser scanner 12, a GPS (Global Positioning System) receiver 14, an azimuth meter 16, a video recording / reproducing device 18, and a data recording / reproducing device 20 are mounted on an automobile. While the vehicle is traveling on the planned road, the video camera 10 captures the road surface in front of the vehicle (or may be behind), and the laser scanner 12 scans the laser beam in the width direction of the road and reflects it from the road surface. Measure the light intensity and the distance to the reflection point. As such a laser scanner 12, what is marketed as a laser ranging apparatus can be utilized. For example, a gyro device can be used for the compass 16. When the gyro device is used, the reception interval of the GPS receiver 14 can be interpolated.

  The GPS receiver 14 continuously measures the vehicle position in the ground coordinate system, and the compass 12 measures north. The direction of the vehicle, that is, the shooting direction of the video camera 10 and the laser irradiation direction of the laser scanner 12 can be known from the output of the direction meter 16. From the ground coordinate value measured by the GPS receiver 14 and the bearing measured by the azimuth meter 16, the ground coordinate corresponding to the road surface of each pixel in the photographing field of the video camera 10 can be determined. Similarly, the ground coordinates of the laser beam reflection point of the laser scanner 12 can also be determined.

  The video recording / reproducing device 18 adds the three-dimensional position coordinate data from the GPS receiver 14 and the azimuth data from the azimuth meter 16 to the output video signal of the video camera 10 and records the result on a recording medium. Further, the data recording / reproducing apparatus 20 converts the reflection data including the distance to the reflection point output from the laser scanner 12, the laser irradiation direction, and the reflected light intensity into the three-dimensional position coordinate data from the GPS receiver 14 and the direction meter 16. Is recorded on the recording medium in association with the azimuth data measured by. In other words, the reflection data obtained by the laser scanner 12 indicates the relative coordinate value of the reflection point and the reflected light intensity viewed from the laser scanner 12.

  Although details will be described later, in this embodiment, the road surface is continuously photographed by the video camera 10 mounted on the vehicle, the line image on the specific horizontal scanning line of the video image is mosaic-combined, and finally the length of the entire road surface is long. A road surface orthoimage observed from the vertical direction with a scale is generated. On the other hand, the laser scanner 12 mounted on the vehicle scans the output laser light in the lateral direction (the width direction of the road) with respect to the road surface, and measures the reflected light intensity. From the measured reflected light intensity, a long reflection intensity image of the road surface along the road is generated. It goes without saying that road surface irregularities can also be measured by taking into account the distance to the reflection point. This reflection intensity image is used as a reference image for removing a shadow portion caused by solar radiation from the road surface orthoimage.

  Since the generated ortho image and the reflected light intensity image are compared at the same position on the road surface, the line image used to generate the ortho image (and hence the mosaic image that is the prerequisite) is scanned by the laser scanner 12. Preferably, it is on a horizontal scan line that roughly corresponds to the line. FIG. 2 shows the positional relationship on the road surface between the photographing range by the video camera 10 and the laser scanning locus by the laser scanner 12. The imaging range of the video camera 10 and the laser scanning line direction of the laser scanner 12 are adjusted in advance so that the laser scanning of the laser scanner 12 falls within the imaging field of view of the video camera 10. At this time, it is preferable that the laser scanning trajectory on the road surface by the laser scanner 12 and the horizontal scanning line for extracting a line image for generating a mosaic image from the video image by the video camera 10 correspond to each other.

  Data recorded by the video recording / reproducing apparatus 18 and the data recording / reproducing apparatus 20 is reproduced and input to the shadow removing apparatus 30. The shadow removal apparatus 30 is actually realized by a shadow removal program that operates on an information processing apparatus including a computer, and part or all of the means other than the storage means constituting the shadow removal apparatus 30 are realized by a computer program. Can be done.

  The ortho image generating device 32 reproduces and reads the video image, the three-dimensional position coordinates by the GPS receiver 14 and the azimuth data by the compass 16 from the video recording / reproducing device 18. The ortho image generating device 32 synthesizes and reconstructs line images on one or more specific horizontal scanning lines of the video image from the video recording / reproducing device 18, and generates a road surface ortho image as viewed from vertically above. The ortho image generating device 32 refers to the three-dimensional position coordinates by the GPS receiver 14 and the azimuth data by the azimuth meter 16, so that the line image on each horizontal scanning line of the video image from the video recording / reproducing device 18 on the road surface. Can determine the ground coordinates. In the generated road surface orthoimage, for example, one pixel has a density of about 2.5 cm. The ortho image generation device 32 stores the generated road surface ortho image data 34 in a storage device.

  The ortho image generation device 32 also outputs coordinate data indicating the ground coordinate range covered by the generated road surface ortho image data as a coordinate file 38. The laser scanner 12 three-dimensionally measures a relatively wide range including the road surface. Among the reflection points measured by the laser scanner 12 using the coordinate file 38, those other than those useful for removing shadows from the road surface image are used. This is because it is excluded. Thereby, subsequent data processing can be lightened.

  The coordinate conversion device 36 reads the reflection data from the laser scanner 12, the three-dimensional position coordinates measured by the GPS receiver 14, and the azimuth data measured by the azimuth meter 16 from the data recording / reproducing device 20. The relative position coordinates of the reflection point viewed from the laser scanner 12 can be determined from the irradiation angle with respect to the road surface of the laser scanner 12, the scanning angle in the width direction, and the azimuth of the azimuth meter 16. By adding the relative position coordinates to the three-dimensional position coordinates measured by the GPS receiver 14, the position of the reflection point in the ground coordinate system can be determined. The coordinate conversion device 36 refers to the coordinate file 38 and extracts only the reflection point data on and around the road surface. The output data of the coordinate conversion device 36 is point cloud data at positions that vary in the ground coordinate system.

The meshing device 40 refers to the pixel coordinates of the road surface orthoimage data 34, and the point cloud data output from the coordinate conversion device 36 is set to the same position as the pixels of the road surface orthoimage data 34 in the ground coordinate system. Convert to point cloud data on mesh or grid. At that time, the reflected light intensity and height are adjusted to numerical values on the mesh or lattice. The resolution of the mesh obtained here is about 2.5 to 10 cm, whereas the resolution of the road surface orthoimage data 34 is about 2.5 cm. Therefore, the mesh interpolation device 42 interpolates the reflection intensity / height of the pixels between the meshes generated by the meshing device 40 so that the same pixel density and arrangement as the road surface orthoimage data 34 are obtained. By this interpolation, point cloud data of reflected light intensity that can be easily compared with the road surface orthoimage data 34 on the ground coordinate system is generated. Obviously, the meshing by the meshing device 40 and the interpolation by the mesh interpolation device 42 may be executed simultaneously.

  The road surface area extraction device 44 extracts a road surface area from the point cloud data output from the mesh interpolation device 42. Specifically, the road surface area extraction device 44 calculates a normal vector for each triangle formed by three adjacent mesh points using a TIN (Triangulated Irregular Network) method used in a numerical terrain model, Whether the road surface is determined by the inclination.

In the example shown in FIG. 3, when P (n−1, m−1) and P (n, m−1) are two points constituting a triangle with respect to the attention point P (n, m), this triangle Is calculated as follows. As shown in FIG. 4, let P (n, m) be the point of interest P1, its x coordinate value be x1, y coordinate value be y1, and z coordinate value be z1. Let P (n-1, m-1) be P2, the x coordinate value is x2, the y coordinate value is y2, and the z coordinate value is z2. Let P (n, m-1) be P3, its x coordinate value is x3, y coordinate value is y3, and z coordinate value is z3. At this time, if the x component, y component, and z component of the normal vector are Nx, Ny, and Nz, respectively,
Nx = (y2-y1) * (z3-z2)-(z2-z1) * (y3-y2)
Ny = (z2-z1) * (x3-x2)-(x2-x1) * (z3-z2)
Nz = (x2-x1) * (y3-y2)-(y2-y1) * (z3-z2)
It becomes.

  FIG. 5 shows a flowchart of an operation for calculating a normal vector for each triangle formed by three adjacent mesh points. The mesh number in the x direction is indicated by n, and the mesh number in the y direction is indicated by m. Assume that there are N + 2 meshes in the x direction, and the variable n is numbered from 0 to N + 1. Further, it is assumed that there are M + 2 in the y direction, and the variable m is numbered from 0 to M + 1.

  First, in order to designate the inside of the outermost mesh, the variable n is initialized with 1 as variables n and m for designating the mesh of interest (S1), and the variable m is initialized with 1 (S2). As shown in FIG. 3, since eight TINs can be formed around the mesh of interest, normal vectors are sequentially clockwise from the eight TINs, for example, in order from the hatched TIN shown in FIG. Calculate (S3 to S6). That is, 1 is substituted into a variable i designating a calculation target TIN (S3), and a normal vector V (n, m, i) of TIN indicated by the variable i is calculated (S4). The variable i is incremented (S5), and steps S4 and S5 are repeated until the variable i exceeds 8 (S6). As a result, the normal vectors V (n, m, i) for the eight TINs surrounding the attention point P (n.m) are calculated.

  The variable m is incremented (S7), and the processes of steps S3 to S7 are repeated until the variable m becomes equal to M (S8). Further, the variable n is incremented (S9), and steps S2 to S9 are repeated until n becomes equal to N (S10).

  According to the flow shown in FIG. 5, the normal vectors of eight surrounding TINs are calculated for all mesh points except the outermost mesh point. In the flow shown in FIG. 5, normal vector calculation by TIN is performed in duplicate on a plane formed by four mesh points adjacent in the horizontal and vertical directions. For example, this can be avoided by setting the increments of n and m to 2. In this case, it is necessary to associate the calculated normal vector data with the mesh points that are not designated as the attention points.

  FIG. 6 shows a road surface determination flowchart for determining whether or not the road surface using the normal vector calculated in the flow shown in FIG.

  On the road surface, the slope of the normal vector is small in principle. Even if there is an inclination, the inclination is small, and the adjacent triangle is inclined in a certain direction. Conversely, the slope of the normal vector is large except on the road surface. If the direction of the normal vector is not constant between adjacent triangles, it is not a road surface because it does not constitute a plane in the first place. The road surface area extraction device 44 sets the road surface flag = 1 when it is determined that the eight triangles formed between the target mesh point and the surrounding eight points are road surfaces under such criteria. Allocation, otherwise road surface flag = 0 is allocated.

  First, in order to designate the inside of the outermost mesh, the variable n is initialized with 1 as variables n and m for designating the mesh of interest (S21), and the variable m is initialized with 1 (S22). As shown in FIG. 3, 1 is substituted into a variable i that designates one of eight TINs around the target mesh (S23).

  It is determined whether or not the angle of the normal vector V (n, m, i) is less than 15 degrees with respect to the vertical (S24). If it is less than 15 degrees, 1 is set to the road surface flag (n, m, i). Substitute (S25). When the angle of the normal vector V (n, m, i) is 15 degrees or more with respect to the vertical (S24), the normal vector V (n, m, i) and the normal vector N of two adjacent TINs It is determined whether or not the angle difference between (n, m, i-1) and V (n, m, i + 1) is less than 20 degrees (S26). If the angle difference with either one is less than 20 degrees (S26), it is determined that the condition is achieved, and 1 is substituted into the road surface flag (n, m, i) (S25). When the angle difference from any normal vector N (n, m, i-1), V (n, m, i + 1) is 20 degrees or more (S26), the road surface flag (n, m, i) is set to 0. Substitute (S27).

  The variable i is incremented (S28), and steps S24 to S28 are repeated until the variable i exceeds 8 (S29). Thereby, it can be temporarily determined whether or not the eight TINs surrounding the attention point P (nm) are road surfaces.

  The variable m is incremented (S30), and the processes of steps S23 to S29 are repeated until the variable m becomes equal to M (S31). Further, the variable n is incremented (S32), and steps S2 to S32 are repeated until n becomes equal to N (S33).

  The road surface area extraction device 44 refers to the road surface flag obtained by the flow shown in FIG. 6 and clusters or groups TINs in which the road surface flag = 1 continues. Since this main clustering method is well known, further explanation is omitted. When a non-road surface exists in the area determined to be a road surface, if the area is less than a predetermined value, it is included in the road surface as a determination error.

  The road surface reflection image generation device 46 generates an 8-bit scale reflection intensity image with the reflected light intensity as a luminance value for the road surface area extracted by the road surface area extraction device 44 and a certain range around it. Ignore extremely low reflected light intensity and extremely high reflected light intensity. A smoothing process for that purpose may be added as a pre-processing for imaging. The road surface reflection image generation device 46 stores the generated road surface reflection intensity image data 48 in a storage device.

  In general, it is preferable that the average reflection intensity value of the road surface reflection intensity image is a level that can be compared with the average luminance value with the road surface ortho image data 34. Therefore, the road surface reflection image generation device 46 reads the RGB luminance data corresponding to the road surface corresponding portion extracted by the road surface area extraction device 44 from the road surface orthoimage data 34, calculates the average luminance value, and reflects the reflection intensity data within a predetermined range. Are assigned to the same scale (for example, 8 bits) as the road surface orthoimage data 34 so that the average value corresponds to the average luminance value of the road surface orthoimage data 34.

  For example, while the brightness value of road surface ortho image data is 8 bits and 0-255, the laser reflected light intensity is very wide such that the value exceeds 0-1000. But 95% or more falls between 200-600. Of course, it varies depending on the performance of the laser scanner 12 and the nature of the reflecting surface.

As an example, the brightness average value of the road surface orthoimage data 34 is 120, and the value that maximizes the statistic of the reflection intensity (uses the maximum number because the average value is affected by a large noise value) is 400. It is assumed that the value that reaches 1% of the total number from 0 is 200. In this case, the reflection intensity value 400 is converted to the value 120, the reflection intensity 200 is converted to the value 1, and the reflection intensity value 200 or less is set to the value 0. And
(Maximum number of reflection intensities−lower limit of reflection intensity) / (average ortho luminance) = ratio (1)
Value of reflection intensity maximum number + (255−luminance average)) × ratio = reflection intensity upper limit (2)
And From equation (1)
(400−200) /120=1.66
From equation (2)
400+ (255-120) × 1.66 = 624
It becomes.

  The value 255 is set for the reflection intensity upper limit (624 in this numerical example) obtained by the equation (2) or more. However, an obvious abnormal value, for example, a reflection intensity of 1000 or more is set to 0.

  Since the road surface reflection intensity image data 48 generated in this way is not affected by the shadow caused by solar radiation on the road surface, it can be used as reference data for separating and extracting the shadow portion on the road surface orthoimage. However, the road surface has white lines for distinguishing manholes and lanes and colored indications of various driving instructions. The reflection intensity of manholes is lower than that of the road surface itself, and the reflection intensity of colored indications such as white lines is that of the road surface itself. Higher than.

  The shadow part recognizing device 50 compares the road surface ortho image data 34 and the road surface reflection intensity image data 48 in pixel units of the same ground coordinate system, recognizes a shadow part on the road surface ortho image, and the shadow part correcting device 52 Then, the color of the recognized shadow part is corrected to be assimilated to the surroundings. FIG. 7 shows an operation flowchart of shadow part recognition processing by the shadow part recognition device 50 and correction processing by the shadow part correction device 52.

  The road surface reflection intensity image data 48 does not include a shadow portion and defines a road surface range, whereas the road surface orthoimage data 34 includes a shadow due to solar radiation, a road shoulder, and surrounding ground objects in addition to the road surface. Therefore, a portion determined to be a road surface based on the road surface reflection intensity image data 48 is separated and extracted from the road surface orthoimage data 34, and a portion that is largely dark in the separated and extracted road surface portion is separated and extracted as a shadow portion.

The shadow part recognition apparatus 50 first reads the road surface orthoimage data 34, acquires the color tone of the entire image, and corrects the color balance (S41). Specifically, in the entire image of the road surface orthoimage data 34, each color of R, G, B is divided into four levels of intensity levels, and the average value is calculated for the two central levels close to the color of the road surface. . For example, if the average luminance of R (red) is C _R , the average luminance of G (green) is C _B , and the average luminance of B (blue) is C _B , the average luminance AV _RGB is
_{AV RGB = (C R + C} G + C B) / 3
It becomes.

The correction coefficient for _R is H _R , the correction coefficient for _G is H _G , and the correction coefficient for _B is H _B.
H _R = C _R / AV _RGB
H _G = C _G / AV _RGB
H _B = C _B / AV _RGB
It becomes. The color balance of the road surface orthoimage data 34 is corrected using the correction coefficients H _R , H _G , and H _B. That is, the R value of the road surface orthoimage data 34 is multiplied correction coefficient H _R, multiplied by the correction coefficient H _G, to G value, multiplied by a correction coefficient H _B to B value. The road surface ortho image data 34 after the color balance adjustment is referred to as color adjusted road surface ortho image data for convenience.

  On the road surface, there are various white and orange colored displays such as manholes and white lines that divide the driving lane. Manholes have a lower reflection intensity than the average road surface, and colored displays have a higher reflection intensity. If the road surface reflection intensity image data 48 and the color-adjusted road surface ortho image data are compared with these, the original road surface and its shadow may be misidentified. Therefore, in this embodiment, a lower limit threshold p1 for excluding manholes from the comparison target and an upper limit threshold p2 for excluding colored display from the comparison targets are set in the shadow part recognition device 50 from the reflection intensity image data 48. Then, a difference threshold value p3 for comparison with the color-adjusted road surface ortho image data is set in the shadow part recognizing device 50 (S42). The difference threshold p3 is set after trial. However, as will be described later, when it is not necessary to exclude the manhole as a comparison target, or in the case of a road where no manhole exists in the first place, the lower limit threshold p1 may be set to zero. Similarly, when it is not necessary to exclude the colored display, or in the case of a road where no colored display exists in the first place, the upper limit threshold p2 is set to the theoretical maximum value of reflection intensity (255 in the case of 8 bits).

  When the reflection intensity of the road surface reflection intensity image data 48 is hierarchized within a predetermined intensity range, it becomes easy to distinguish the road surface from the features (manholes, road signs printed on the road surface, white lines, orange lines, etc.). By this hierarchization, the road surface itself and the feature are identified, and the lower limit threshold p1 and the upper limit threshold p2 are selected or set for the feature to be excluded. Thereby, the lower limit threshold p1 and the upper limit threshold p2 can be set appropriately.

  For example, in the case of 8 bits, when layering in an intensity range having a reflection intensity value of about 20, the reflection intensity of the road surface reflection intensity image data 48 is hierarchized or grouped in about 13 levels by 225/20. Areas that belong to the hierarchy around the average reflection intensity indicate the road surface itself, and among the remaining areas, manholes exist in the low reflection intensity hierarchy, and white lines, orange lines, and roads printed on the high reflection intensity hierarchy There will be signs and the like. In one example, the reflection intensity of the road surface itself was about 134, that of the manhole was about 97, that of the white stop line was about 226, and the yellow road sign was about 181. In this case, when it is desired to exclude the manhole from the comparison target, the lower limit threshold p1 may be set to an intermediate value between 97 and 134, for example, about 110. When it is desired to exclude the yellow road sign from the comparison target, the upper limit threshold p2 may be set to an intermediate value between 134 and 181, for example, about 160.

The shadow part recognition apparatus 50 uses the set threshold values p1, p2, and p3, compares the RGB intensity with the reflection intensity of the road surface reflection intensity image data 48 and the color-adjusted road surface ortho image data, and determines the shadow part. The pixel unit is specified (S43). The shadow part recognition apparatus 50 first extracts pixels to be used for comparison with the color-adjusted road surface ortho image data from the road surface reflection intensity image data 48 using the threshold values p1 and p2. That is, ir (x, y) is the reflection intensity of the pixel (x, y), and averbb (x, y) is the RGB average value of the pixel (x, y) obtained from the color-adjusted road surface ortho and image data. And The shadow part recognizing device 50 corresponds to the threshold values p1 and p2.
p1 <ir (x, y) <p2
A pixel that satisfies the above condition is to be compared with color-adjusted road surface ortho image data. The shadow part recognition apparatus 50 determines whether the difference between the reflection intensity ir (x, y) satisfying the above expression and the RGB average value averageb (x, y) exceeds the threshold value p3 or not, and the pixel is a shadow pixel. That is, the shadow part recognition device 50
ir (x, y) -avergb (x, y)> p3
Recognize a pixel that holds as a shadow.

By the way, when the reflection intensity of the pixel corresponding to the manhole is also compared with the RGB average value of the color-adjusted road surface ortho image data,
ir (x, y) <p2
as well as,
ir (x, y) -avergb (x, y)> p3
Is a condition for establishing a shadow.

When the reflection intensity of the pixel corresponding to the coloring display is compared with the RGB average value of the color-adjusted road orthoimage data,
p1 <ir (x, y)
as well as,
ir (x, y) -avergb (x, y)> p3
Is a condition for establishing a shadow.

In both manholes and colored displays, when the reflection intensity of the corresponding pixel is compared with the RGB average value of the color-adjusted road surface ortho image data,
ir (x, y) -avergb (x, y)> p3
Is a condition for establishing a shadow.

  The shadow part recognizing device 50 groups the pixels identified as shadow parts in step S42 by adjacent pixels (S44). At the time of grouping, groups that match within 30% of average luminance are integrated as the same shadow portion, and conversely, a group that exceeds 30% and has different average luminance is set as another shadow group. Thereby, each shadow part is specified as a shadow area.

  More specifically, for example, a search for shadow pixels is started from the upper left of the image. The shadow pixel that appears first is set as the starting point of grouping, and the luminance value of the starting point is set as the reference luminance value of the shadow group. A group search is performed on pixels around the starting point, and shadow pixels that do not belong to any shadow group and have a difference of 30% or less from the reference luminance value are set as the same shadow group. The average luminance value is recalculated by adding the luminance value of the shadow pixel added to the shadow group, and set as the reference luminance value. Such a region search is recursively performed to determine the shadow group region. After the determination, shadow pixels that are not grouped are searched again. If there is a shadow pixel that does not belong to any shadow group, shadow grouping is executed as described above. The above processing is executed toward the lower right of the image. The search direction itself may be from the upper left to the lower right or vice versa, and it is sufficient to search exhaustively within the image. Such a technique for grouping (or clustering) a large number of pixels with the same attribute has been adopted in various fields including the field of maps, and various methods known also in this embodiment. Can be used.

  Next, the shadow partial correction device 52 corrects the shadow area to a color tone when there is no shadow (S45 to S48). For this purpose, first, the shadow correction device 52 obtains the RGB data of the peripheral area from the color-adjusted road surface ortho image data for each shadow area (shadow group) specified in step S44 (S45). Apply (S46). Specifically, the shadow partial correction device 52 reads out the RGB values of the peripheral region having a certain width for each shadow region from the color-adjusted road surface ortho image data, and calculates the average intensity values of the R signal, the G signal, and the B signal. Calculate (S45). However, the RGB values of pixels having a difference of 20% or more in intensity and pixels having a RGB balance deviation of 15% or more are excluded from the average intensity calculation target. The shadow partial correction device 52 calculates the intensity values of the R signal, the G signal, and the B signal of the shadow area so as to coincide with the average intensity values of the R signal, the G signal, and the B signal of the peripheral area calculated in this way. Is adjusted (S46).

  As a result of adjusting the intensity values of the R signal, the G signal, and the B signal in the shadow area, there is a step in the intensity value of each color component at the boundary between the shadow area and the surrounding non-shadow area. Therefore, the shadow part correcting device 52 adjusts (smoothing) the intensity value of each color component in the boundary part with the shadow area on the non-shadow area side in contact with each shadow area (S47).

  FIG. 8 is a schematic diagram for explaining the shadow portion correction process. FIG. 8A shows a change in luminance (intensity of each color component) in the road surface direction. The intensity decreases from the non-shadow area toward the shadow area. FIG. 8B shows the shadow area brightness value (intensity value) so that the shadow partial correction device 52 matches the average brightness value (average intensity value) of the non-shadow area adjacent to the shadow area in step S46. Increased results are shown. As a result of the processing in step S46, as shown in FIG. 8B, a step in intensity occurs at the boundary portion on the non-shadow area side adjacent to the shadow area. The shadow part correcting device 52 smoothes the intensity step at the boundary between the intensity of the shadow area located on the both sides and the intensity of the non-shadow area (S47).

  The shadow part correction device 52 further executes a color detail correction process for reproducing the details of the color information for the shadow part (S48). Simply correcting the color tone of the shadow area with the surrounding color information will make the color of the shadow area uniform. On the other hand, the road surface reflection intensity image data 48 has a low resolution of the assumed laser reflection intensity. Therefore, even if the road surface reflection intensity image data 48 or the assumed laser reflection intensity is used for color tone correction of the shadow area, Color becomes uniform. Therefore, in this embodiment, detailed color information is acquired from the color-adjusted road surface orthoimage data (or road surface orthoimage data 34) before color correction in steps S45 and S46, and the color intensity of the pixel of interest is corrected. did.

  A specific processing method will be described. The shadow portion correction device 52 sequentially selects arbitrary pixels in the shadow area from the road surface orthoimage data processed in step S47 and sets it as the target pixel. Next, the shadow partial correction device 52 uses the color-adjusted road surface ortho image data (or the road surface ortho image data 34) for each color intensity of RGB of the shadow pixel in a predetermined range (for example, 5 × 5 pixels) centered on the target pixel. The intensity average value is calculated for each color. For each color, a result obtained by subtracting the intensity average value calculated from the color intensity of the pixel of interest from the color intensity of the pixel of interest is used as a correction value, and is added to the road surface ortho image data after the processing in step S47. The above processing is executed for all shadow pixels of the road surface ortho image data processed in step S47.

  By this color detail correction processing, the color gradation of the shadow portion in the video image can be reproduced after removing the shadow.

  The shadow partial correction device 52 stores the road surface ortho image data processed as described above, that is, the shadow removal road surface ortho image data 54 in the storage device.

  In this way, according to the present embodiment, by using the road surface orthoimage obtained by actual shooting and the reflection intensity data obtained by the laser scanner that is not affected by the shadow caused by solar radiation, the shadow portion of the road surface portion can be obtained from the road surface orthoimage obtained by actual shooting. Can be removed appropriately.

  In the above embodiment, a laser scanner is used to acquire a road surface image that is not affected by the shadow caused by solar radiation. However, since the image obtained by the infrared camera is also not affected by the shadow caused by the solar radiation, the road surface orthoimage by visible light is used. The data 34 can be used as a reference image. That is, an image obtained by an infrared camera can be used as an alternative to reflection intensity data by a laser scanner, and can be used as a reference image of the road surface ortho image data 34 through mosaic synthesis, meshing, and road surface area extraction.

  In the above embodiment, the pixel coordinates of the road surface orthoimage data 34 are determined first, and then the meshing device 40 refers to the road surface orthoimage data 34 to determine the pixel coordinates of the point cloud data of the reflection intensity. However, the pixel coordinates of the road surface orthoimage data 34 and the road surface reflection intensity image data 48 may be determined in the reverse order. Furthermore, separately generated pixel coordinates (and resolution), road surface orthoimage data and road surface reflection intensity image data 48 having the same pixel coordinates may be generated.

  By adding the color data of the shadow removal road surface orthoimage data 54 obtained in the first embodiment to the road surface reflection intensity image data 48, it is possible to generate colored three-dimensional point cloud data. In this case, the shadow part correction process (S45 to S48) by the shadow part correction device 52 is applied to the road surface reflection intensity image data 48 as a target.

  Further, by adding the color data of the shadow removal road surface ortho image data 54 to the interpolated point cloud data output from the mesh interpolation device 42 in the same manner as the road surface reflection intensity image data 48, three-dimensional data with color information is added. Point cloud data can be generated.

  Although the invention has been described with reference to specific illustrative embodiments, various modifications and alterations may be made to the above-described embodiments without departing from the scope of the invention as defined in the claims. This is obvious to an engineer in the field to which the present invention belongs, and such changes and modifications are also included in the technical scope of the present invention.

10: Video camera 12: Laser scanner 14: GPS (Global Positioning System) receiver 16: Direction meter 18: Video recording / reproducing device 20: Data recording / reproducing device 30: Shadow removing device 32: Ortho image generating device 34: Road surface ortho image Data 36: Coordinate conversion device 38: Coordinate file 40: Meshing device 42: Mesh interpolation device 44: Road surface area extraction device 46: Road surface reflection image generation device 48: Road surface reflection intensity image data 50: Shadow portion recognition device 52: Shadow portion Correction device 54: Shadow removal road surface ortho image data

Claims (32)

An imaging device (10) for photographing the road surface in a shaded state due to solar radiation;
Road surface ortho image generation means (32) for generating road surface ortho image data (34) from a plurality of image data obtained by the imaging device;
A laser scanner (12) that scans the same road surface with laser light and measures the reflection intensity at the reflection point;
Reference image generation means for generating road surface reflection intensity image data (48) from the reflection intensity data of the reflection point obtained by the laser scanner as reference image data for comparison with the road surface ortho image data in a prescribed reflection intensity range ( 36-46),
A shadow part recognizing means (50) for recognizing a shadow area of the road surface ortho image data with reference to the reference image data;
Shadow part correction means (52) for correcting each shadow area recognized by the shadow part recognition means with the color information of the peripheral part of the shadow area of the road surface orthoimage data
A road surface image generation system comprising:   The road surface image generation system according to claim 1, wherein the road surface ortho image data (34) and the road surface reflection intensity image data (48) have the same pixel coordinates. The reference image generating means
Mesh data generating means (36, 40, 42) for generating mesh data having the same pixel coordinates as the road surface orthoimage data from the reflection intensity data of the reflection point;
Road surface area extracting means (44) for extracting a road surface area from the mesh data;
Road surface reflection intensity image generation means (46) for generating the road surface reflection intensity image data (48) from the mesh data extracted by the road surface area extraction means.
The road surface image generation system according to claim 1, further comprising: The shadow part recognition means is
A shadow specifying means (S43) for specifying a shadow pixel constituting a shadow by comparing the road surface ortho image data and the reference image data in units of pixels;
Shadow grouping means for grouping shadow pixels specified by the shadow specifying means into shadow areas (S44)
The road surface image generation system according to any one of claims 1 to 3, further comprising: The shadow part recognizing means further comprises color balance adjusting means (S41) for adjusting the color balance of the road surface ortho image data before the shadow specifying by the shadow specifying means,
The shadow specifying unit specifies a shadow pixel constituting a shadow by comparing the road surface ortho image data color-adjusted by the color balance adjusting unit and the reference image data in units of pixels. Item 5. The road surface image generation system according to Item 4. The shadow part recognition means sets a lower threshold (p1) and an upper threshold (p2) for the reference image data, and a threshold for setting a difference threshold (p3) for an intensity difference between the road orthoimage data and the reference image data. Setting means (S42),
The shadow specifying means extracts pixels having an intensity larger than the lower limit threshold and smaller than the upper limit threshold from the reference image data, and the difference between the intensity and the color average intensity of the road surface ortho image data among the extracted pixels. The road surface image generation system according to claim 4 or 5, wherein a pixel that exceeds the difference threshold (p3) is identified as the shadow pixel. The shadow part correcting means is
Color adjusting means (S45, S46) for adjusting the color intensity of each shadow region recognized by the shadow part recognizing means according to the surrounding color information on the road surface orthoimage data;
Smoothing means (S47) for smoothing the color change of the shadow area adjusted by the color adjustment means and its peripheral portion;
A means for reproducing the details of the color information of the shadow area, wherein the color intensity of the pixel of interest in the shadow area is a difference value from the average intensity of a predetermined number of surrounding pixels of the pixel of interest on the road surface orthoimage data The road surface image generation system according to any one of claims 1 to 6, further comprising detail reproduction means that corrects the value according to   Further, it comprises means for adding the color data of the corrected road surface ortho image data obtained by the shadow portion correcting means (52) to the road surface reflection intensity image data, thereby generating three-dimensional point cloud data. The road surface image generation system according to any one of claims 1 to 7. A shadow removing device that removes the shadow from image data obtained by photographing a road surface with a shadow caused by solar radiation,
Road surface ortho image generating means (32) for generating road surface ortho image data (34) from the image data obtained by photographing the road surface;
A reflection intensity range defined as reference image data for comparing road surface reflection intensity image data (48) with the road surface ortho image data from reflection intensity data measured by scanning the same road surface with laser light with a laser scanner. Reference image generation means (36 to 46) generated in
A shadow part recognizing means (50) for recognizing a shadow area of the road surface ortho image data with reference to the reference image data;
Shadow part correction means (52) for correcting each shadow area recognized by the shadow part recognition means with the color information of the peripheral part of the shadow area of the road surface orthoimage data
A shadow removing apparatus comprising:   10. The shadow removing device according to claim 9, wherein the road surface ortho image data (34) and the road surface reflection intensity image data (48) have the same pixel coordinates. The reference image generating means
Mesh data generating means (36, 40, 42) for generating mesh data having the same pixel coordinates as the road surface orthoimage data from the reflection intensity data of the reflection point;
Road surface area extracting means (44) for extracting a road surface area from the mesh data;
Road surface reflection intensity image generation means (46) for generating the road surface reflection intensity image data (48) from the mesh data extracted by the road surface area extraction means.
The shadow removing apparatus according to claim 9, further comprising: The shadow part recognition means is
A shadow specifying means (S43) for specifying a shadow pixel constituting a shadow by comparing the road surface ortho image data and the reference image data in units of pixels;
Shadow grouping means for grouping shadow pixels specified by the shadow specifying means into shadow areas (S44)
The shadow removing apparatus according to claim 9, further comprising: The shadow part recognizing means further comprises color balance adjusting means (S41) for adjusting the color balance of the road surface ortho image data before the shadow specifying by the shadow specifying means,
The shadow specifying unit specifies a shadow pixel constituting a shadow by comparing the road surface ortho image data color-adjusted by the color balance adjusting unit and the reference image data in units of pixels. Item 13. The shadow removing device according to Item 12. The shadow part recognition means sets a lower threshold (p1) and an upper threshold (p2) for the reference image data, and a threshold for setting a difference threshold (p3) for an intensity difference between the road orthoimage data and the reference image data. Setting means (S42),
The shadow specifying means extracts pixels having an intensity larger than the lower limit threshold and smaller than the upper limit threshold from the reference image data, and the difference between the intensity and the color average intensity of the road surface ortho image data among the extracted pixels. The shadow removing apparatus according to claim 12 or 13, wherein a pixel that exceeds the difference threshold (p3) is identified as the shadow pixel. The shadow part correcting means is
Color adjusting means (S45, S46) for adjusting the color intensity of each shadow area recognized by the shadow part recognizing means according to the surrounding color information on the ortho image data;
Smoothing means (S47) for smoothing the color change of the shadow area adjusted by the color adjustment means and its peripheral portion;
A means for reproducing the details of the color information of the shadow area, wherein the color intensity of the pixel of interest in the shadow area is a difference value from the average intensity of a predetermined number of surrounding pixels of the pixel of interest on the road surface orthoimage data The shadow removal apparatus according to claim 9, further comprising a detail reproduction unit that corrects the value according to the value according to.   Further, it comprises means for adding the color data of the corrected road surface ortho image data obtained by the shadow portion correcting means (52) to the road surface reflection intensity image data, thereby generating three-dimensional point cloud data. The shadow removing apparatus according to claim 9, wherein: A shadow removal method for removing the shadow from image data obtained by photographing a road surface with a shadow caused by solar radiation,
Replaying road surface video data obtained by photographing the road surface with a video camera from a recording medium and loading it into an information processing device;
A road surface ortho image data generating step (32) in which the road surface ortho image generating means (32) of the information processing device generates road surface ortho image data (34) from the road surface video data;
The reference image generation means (36 to 46) of the information processing apparatus obtains road surface reflection intensity image data (48) from the reflection intensity data measured by scanning the same road surface with a laser beam with a laser scanner. A reference image generation step for generating a reference image data for comparison with image data in a prescribed reflection intensity range;
A shadow part recognition step in which the shadow part recognition means (50) of the information processing apparatus refers to the reference image data and recognizes a shadow area of the road surface ortho image data;
The shadow part correction means (52) of the information processing apparatus corrects each shadow area recognized by the shadow part recognition means with the color information of the peripheral part of the shadow area of the road surface orthoimage data. A shadow removal method comprising: steps.   18. The shadow removal method according to claim 17, wherein the road surface ortho image data (34) and the road surface reflection intensity image data (48) have the same pixel coordinates. The reference image generation step includes
A mesh data generating unit (36, 40, 42) of the information processing apparatus for generating mesh data having the same pixel coordinates as the road surface ortho image data from the reflection intensity data of the reflection point;
A road surface area extracting unit (44) of the information processing apparatus extracts a road surface area from the mesh data;
The road surface reflection intensity image generation unit (46) of the information processing apparatus includes a road surface reflection intensity image generation step of generating the road surface reflection intensity image data (48) from the mesh data extracted by the road surface area extraction unit. The shadow removal method according to claim 17, wherein: The shadow part recognition step includes
A shadow specifying step (S43) for specifying a shadow pixel constituting a shadow by comparing the road surface ortho image data and the reference image data in pixel units;
Shadow grouping step for grouping shadow pixels specified by the shadow specifying means into shadow areas (S44)
The shadow removal method according to claim 17, further comprising: The shadow part recognition step further includes a color balance adjustment step (S41) for adjusting the color balance of the road surface ortho image data before the shadow specification by the shadow specification step,
The shadow specifying step specifies a shadow pixel constituting the shadow by comparing the road surface ortho image data color-adjusted in the color balance adjusting step and the reference image data in pixel units. Item 20. The shadow removal method according to Item 20. The shadow part recognition step includes a threshold value for setting a lower limit threshold value (p1) and an upper limit threshold value (p2) for the reference image data, and a difference threshold value (p3) for an intensity difference between the road surface orthoimage data and the reference image data. A setting step (S42),
The shadow specifying step extracts pixels having an intensity larger than the lower limit threshold and smaller than the upper limit threshold from the reference image data, and the difference between the intensity and the color average intensity of the road surface ortho image data among the extracted pixels. The shadow removal method according to claim 20 or 21, wherein a pixel that exceeds the difference threshold (p3) is identified as the shadow pixel. The shadow part correction step includes
A color adjustment step (S45, S46) for adjusting the color intensity of each shadow region recognized by the shadow part recognition means in accordance with the surrounding color information on the ortho image data;
A smoothing step (S47) for smoothing the color change of the shadow area and its peripheral part adjusted in the color adjustment step;
A step of reproducing details of the color information of the shadow region, wherein the color intensity of the pixel of interest in the shadow region is different from the average intensity of a predetermined number of surrounding pixels of the pixel of interest on the road surface ortho image data The shadow removal method according to any one of claims 17 to 22, further comprising a detail reproduction step of correcting with a value according to.   Further, the information processing apparatus adds the color data of the corrected road surface ortho image data obtained by the shadow portion correcting means (52) to the road surface reflection intensity image data, thereby generating three-dimensional point cloud data. The shadow removal method according to any one of claims 17 to 23, comprising: In order to remove the shadow from the image data obtained by photographing the road surface in the shaded state due to solar radiation,
Road surface ortho image generating means (32) for generating road surface ortho image data (34) from the image data obtained by photographing the road surface;
A reflection intensity range defined as reference image data for comparing road surface reflection intensity image data (48) with the road surface ortho image data from reflection intensity data measured by scanning the same road surface with laser light with a laser scanner. Reference image generation means (36 to 46) generated in
A shadow part recognizing means (50) for recognizing a shadow area of the road surface ortho image data with reference to the reference image data;
Shadow part correction means (52) for correcting each shadow area recognized by the shadow part recognition means with the color information of the peripheral part of the shadow area of the road surface orthoimage data
Shadow removal program to function as   26. The shadow removal program according to claim 25, wherein the road surface ortho image data (34) and the road surface reflection intensity image data (48) have the same pixel coordinates. The reference image generating means
Mesh data generating means (36, 40, 42) for generating mesh data having the same pixel coordinates as the road surface orthoimage data from the reflection intensity data of the reflection point;
Road surface area extracting means (44) for extracting a road surface area from the mesh data;
Road surface reflection intensity image generation means (46) for generating the road surface reflection intensity image data (48) from the mesh data extracted by the road surface area extraction means.
26. The shadow removal program according to claim 25, comprising: The shadow part recognition means is
A shadow specifying means (S43) for specifying a shadow pixel constituting a shadow by comparing the road surface ortho image data and the reference image data in units of pixels;
Shadow grouping means for grouping shadow pixels specified by the shadow specifying means into shadow areas (S44)
The shadow removal program according to any one of claims 25 to 27, further comprising: The shadow part recognizing means further comprises color balance adjusting means (S41) for adjusting the color balance of the road surface ortho image data before the shadow specifying by the shadow specifying means,
The shadow specifying unit specifies a shadow pixel constituting a shadow by comparing the road surface ortho image data color-adjusted by the color balance adjusting unit and the reference image data in units of pixels. Item 29. The shadow removal program according to Item 28. The shadow part recognition means sets a lower threshold (p1) and an upper threshold (p2) for the reference image data, and a threshold for setting a difference threshold (p3) for an intensity difference between the road orthoimage data and the reference image data. Setting means (S42),
The shadow specifying means extracts pixels having an intensity larger than the lower limit threshold and smaller than the upper limit threshold from the reference image data, and the difference between the intensity and the color average intensity of the road surface ortho image data among the extracted pixels. 30. The shadow removal program according to claim 28 or 29, wherein a pixel that exceeds the difference threshold (p3) is identified as the shadow pixel. The shadow part correcting means is
Color adjusting means (S45, S46) for adjusting the color intensity of each shadow area recognized by the shadow part recognizing means according to the surrounding color information on the ortho image data;
Smoothing means (S47) for smoothing the color change of the shadow area adjusted by the color adjustment means and its peripheral portion;
A means for reproducing the details of the color information of the shadow area, wherein the color intensity of the pixel of interest in the shadow area is a difference value from the average intensity of a predetermined number of surrounding pixels of the pixel of interest on the road surface orthoimage data The shadow removal program according to any one of claims 25 to 30, further comprising detail reproduction means for correcting with a value corresponding to.   Further, it comprises means for adding the color data of the corrected road surface ortho image data obtained by the shadow portion correcting means (52) to the road surface reflection intensity image data, thereby generating three-dimensional point cloud data. The shadow removal program according to any one of claims 25 to 31.

Images