JP2007232652A - Device and method for determining road surface condition - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely determine a road surface condition over a wide range, using a detailed classification. <P>SOLUTION: This method/device detects respectively a snow condition based on an ultraviolet image, a water condition based on an infrared image, and freezing based on a temperature distribution image, and determines a comprehensive condition of a road surface based on a combination thereof. The snow and water conditions are detected using a ratio of a pixel value in the sky to that on the road surface, and a ratio of a pixel value in the infrared image to that on the ultraviolet image, in order to avoid an influence of a disturbance. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for determining a road surface state based on an image.

  In cold regions in winter, slip accidents frequently occur due to road freezing. By detecting dangerous road surface conditions such as freezing with a road surface condition sensor (hereinafter referred to as “road surface sensor”) that grasps the road surface conditions (dry, wet, water film, snow pressure, freezing, etc.) and providing them to drivers and vehicles In addition, the driver and the vehicle can perform risk avoidance control such as deceleration to avoid a slip accident.

  Further, in winter cold areas, road managers need to perform snow removal work and antifreeze spraying work appropriately according to the road surface condition. If the road surface sensor is utilized for road management work, the road surface can be monitored in real time, and road management work can be immediately performed when a dangerous event such as freezing occurs. As a result, the road management work can be performed with high quality, and the workload of the road manager can be reduced.

Therefore, many road surface sensors that grasp the state of the road surface by performing image processing on the camera video have been developed, for example, as follows.
The system described in Patent Document 1 irradiates the road surface with infrared light having different absorption rates between water and ice using a light source with a narrow wavelength band such as a semiconductor laser, and receives reflected light from the road surface with an infrared sensor. And discriminate water and ice on the road surface. In addition, light in a wavelength band that is difficult to be absorbed by both water and ice is used in combination to determine whether it is dry, frozen, wet, or snowy.

  The apparatus described in Patent Document 2 irradiates the road surface with a halogen lamp or the like, and detects the degree of mirroring of the road surface based on the brightness value of the reflected light in the region where the light is regularly reflected and other regions, The occurrence of “smooth road surface” is detected. The slippery road surface is a road surface in which the surface of ice or snow on the road surface is mirrored by being polished by a traveling vehicle, and is very slippery and dangerous.

  In the apparatus described in Patent Document 3, an image obtained by imaging a road surface with a visible light camera in advance is made into a database according to the classification of road surface conditions (dry, wet, snow, etc.) and environmental conditions (clear weather, cloudy, nighttime, etc.). Then, it is determined by using the feature amount calculated by statistical analysis which image in the database is closest to the image of the road surface captured by the visible light camera.

  In addition to these, a sensor that determines the road surface condition by analyzing the image obtained through the polarizing filter attached to the front of the camera, utilizing the property that visible light is polarized by the water surface, and infrared rays are specularly reflected by the water surface. A sensor that determines the road surface condition by calculating the specular reflection intensity from an image obtained by capturing infrared reflected light with a camera has been developed. Further, as a road surface freezing determination method common to these methods, a determination method has been developed in which a road surface temperature at a certain location is measured and the road surface state is frozen when the road surface temperature is lower than a predetermined temperature.

However, the conventional road surface sensor has the following problems.
The system of Patent Document 1 uses a semiconductor laser and cannot determine a wide range of road surface conditions. The device of Patent Document 2 specializes in determining a smooth road surface, and cannot determine the road surface state according to fine classifications such as dry, wet, water film, snow pressure, and freezing. The device of Patent Document 3 is troublesome because it is necessary to create a database for each location where the sensor is installed, for each road surface condition, and for each environmental condition. When the database is incomplete, it is caused by disturbances such as surrounding shadows and changes in sunlight. There was a problem that erroneous determination was likely to occur.

  In addition, the method using a polarizing filter has a problem that, in principle, it can only detect moisture, and the method of analyzing the specular reflection intensity of infrared rays can determine the road surface state only in a narrow region where illumination can reach. There was a problem. Also, in the freezing determination method based on the road surface temperature measured at a specific location, if the road surface temperature at the measurement location is higher than the road surface temperature at the location that is actually frozen, it is erroneously determined that it is not frozen. There was a problem.

The road surface sensor required for safe driving support and road management business support applications can detect a wide range of road surface conditions, and further, the road surface state can be detected for each region by dividing the road surface into several regions. That is, there are few misjudgments, initial costs and adjustment costs are small, and a subdivided state such as dry / wet / water film / pressed snow / freeze is known. However, there is currently no road surface sensor that satisfies all these requirements.
JP-A-9-318766 JP 2004-157073 A JP 2002-162343 A

  An object of the present invention is to detect a wide range of road surface conditions so that the user can understand the subdivided road surface state for each area obtained by dividing the road surface, and there are few misjudgments, initial costs, and operation costs. It is to provide a road surface condition determination technique with as little as possible.

  FIG. 1 is a diagram showing the principle of the present invention. The road surface condition determination device 1 according to the present invention uses a UV image, an infrared image, and a temperature distribution image obtained by capturing the road surface in a state where light including ultraviolet rays and infrared rays is applied to the road surface whose state is to be determined. Determine. The road surface condition determination apparatus 1 includes an ultraviolet feature amount calculating unit 2 that calculates an ultraviolet feature amount on the road surface based on the ultraviolet image, an infrared feature amount calculating unit 3 that calculates an infrared feature amount on the road surface based on the infrared image, and a calculation. Road surface state determining means 4 for comprehensively determining the road surface state based on the ultraviolet characteristic amount, the infrared characteristic amount, and the temperature distribution image.

  Hereinafter, the road surface state refers to the state of snow, water, ice on the road surface, and their distribution. It is known that ultraviolet rays are easily scattered when hitting a substance, and most of the ultraviolet rays are scattered particularly on the snow surface. Therefore, it is possible to detect the state of snow on the road surface from the values of feature quantities (intensity, reflectance, etc.) related to ultraviolet rays on the road surface, and to detect the distribution of snow on the road surface if the entire road surface is detected. Can do. In addition, since infrared rays are easily absorbed by water, it is possible to detect the state of water on the road surface from the values of infrared features (intensity, absorption rate, etc.) on the road surface. The distribution of water above can be detected. In addition, the ice condition (freezing condition) on the road surface has a correlation with the road surface temperature. Therefore, the road surface state determination apparatus 1 according to the present invention calculates the values of feature quantities of ultraviolet rays and infrared rays from the image, obtains the road surface temperature distribution from the image, and determines the road surface state.

  The first preferred method for calculating the ultraviolet feature amount in the ultraviolet feature amount calculating means 2 uses an ultraviolet image captured with a composition including the sky and the road surface, and corresponds to the sky pixel value corresponding to the sky region and the road surface region. In this method, the road surface pixel value is calculated, and the ultraviolet feature amount is calculated based on the ratio of the sky pixel value and the road surface pixel value. In the second preferred method, the visible light image is used to calculate the ultraviolet feature amount based on the pixel value of the visible light image and the pixel value of the ultraviolet image, or based on the pixel value of the infrared image and the pixel value of the ultraviolet image. This is a method for calculating the ultraviolet feature amount.

  When the UV intensity on the road surface is used as the UV feature quantity, it is affected by disturbance (season, time zone, shade or sunny). However, in the first method described above, since the ultraviolet feature amount is calculated based on the ratio of the sky pixel value and the road surface pixel value, even if the ultraviolet radiation amount itself varies depending on the season and time zone, it is affected by the influence. It can be determined without. In addition, in the second method described above, in consideration of the fact that not only ultraviolet rays but also the amount of solar radiation of visible rays and infrared rays is reduced in the shade, the feature amount of ultraviolet rays is calculated based on the ratio of the visible light image or the pixel value of the infrared image. Therefore, the influence of whether it is shade or hinata is small.

  A preferred first method for calculating the infrared feature amount in the infrared feature amount calculation means 3 is to use an infrared image captured with a composition including the sky and the road surface, and to correspond to the sky pixel value corresponding to the sky region and the road surface region. The road surface pixel value is calculated, and the infrared feature amount is calculated based on the ratio between the sky pixel value and the road surface pixel value. In the second preferred method, an infrared feature amount is calculated based on the pixel value of the visible light image and the pixel value of the infrared image using the visible light image, or based on the pixel value of the ultraviolet image and the pixel value of the infrared image. This is a method of calculating the infrared feature amount.

  When the infrared intensity on the road surface is used as the infrared feature amount, it is affected by disturbances (season, time zone, shade or sunny). However, in the first method described above, since the infrared feature amount is calculated based on the ratio of the sky pixel value and the road surface pixel value, even if the infrared radiation amount itself varies depending on the season and time zone, it is affected by the influence. It can be determined without. In the second method, the infrared feature amount is calculated based on the ratio of the visible light image or the pixel value of the ultraviolet image in view of the fact that not only the infrared rays but also the amount of solar radiation of the visible rays and ultraviolet rays is reduced in the shade. Therefore, the influence of whether it is shade or hinata is small.

  In addition, if the light utilized by the present invention is light including ultraviolet rays and infrared rays, the reflected light (natural light) from the road surface of sunlight, the light from a car headlight or a projector, the illumination light already installed on the road, etc. Any light such as light reflected by the road surface may be used. The light irradiating the road surface is not limited to the above-mentioned light, and any light can be used because the correction value can be calculated later if the wavelength distribution is known.

  Since the road surface state determination apparatus 1 according to the present invention determines the road surface state by combining three types of information of ultraviolet rays, infrared rays, and temperature, the road surface state can be classified and determined in detail. In addition, since a dedicated laser beam or the like is not required, it is possible to image a wide range of road surfaces and determine the road surface state for the wide range of road surfaces. In addition, since the road surface state is determined from the image, it is possible to determine the entire captured road surface instead of determining the state only for a specific location. The road surface state is divided into small regions by dividing the road surface into small regions. It is also possible to determine. Furthermore, by using a calculation method for eliminating the influence of disturbances in the ultraviolet feature quantity calculation means 2 and the infrared feature quantity calculation means 3, a high determination can be made without performing parameter adjustment or the like according to the season or time zone. Accuracy is obtained. Also, it requires less effort than the database method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 2 is a functional block diagram of the road surface state determination apparatus according to the first embodiment of the present invention. In the first embodiment, an ultraviolet image, an infrared image, and a temperature distribution image are captured using different cameras, and the road surface state is determined based on the captured images. In the first embodiment, since it is necessary for subsequent processing, the image is captured with a composition including not only the road surface whose state is to be determined but also the sky. In the present embodiment, the ultraviolet reflectance is used as the ultraviolet feature quantity, and the infrared absorption rate is used as the infrared feature quantity. In this embodiment, a camera that captures a visible light image is further used. Although the visible light image is not an essential element for carrying out the present invention, the second preferred method is adopted as the method for calculating the infrared feature amount in the present embodiment, so a visible light image is necessary. .

  The road surface state determination apparatus according to the first embodiment includes an ultraviolet camera imaging unit 101, an ultraviolet image generation unit 102, a road surface / sky region extraction unit 103, an ultraviolet reflectance calculation unit 104, a light projection determination unit 105, and a visible light camera imaging unit. 106, visible light image generation unit 107, first road surface region extraction unit 108, infrared camera imaging unit 109, infrared image generation unit 110, second road surface region extraction unit 111, infrared absorptance calculation unit 112, infrared thermography imaging unit 113, a road surface state determination unit 114, a road surface state image generation unit 115, a road surface determination result output unit 116, and a temperature distribution image generation unit 117.

  The ultraviolet camera imaging unit 101 captures the road surface and the sky together, and the ultraviolet image generation unit 102 generates an ultraviolet image as digital data. From the generated ultraviolet image, a road surface / sky region extraction unit 103 extracts a road surface region and a celestial sphere region. For example, the ultraviolet camera imaging unit 101 and the ultraviolet image generation unit 102 may use a known configuration as an ultraviolet CCD (Charge Coupled Device) camera.

  For example, the following various methods can be used as the road surface area and sky area extraction method in the road surface / sky area extraction unit 103. A white line on the road surface may be detected, and a range surrounded by the white line may be used as the road surface. It is possible to determine which area of the photographed image is on the road surface and which area is in the sky based on the installation position and orientation of the camera. Since a large amount of ultraviolet light is contained in the sky, a place where the pixel value is larger than a predetermined value in the ultraviolet image may be the sky. When the sky is imaged with an infrared thermography apparatus, it is recognized that infrared radiation is almost zero. Therefore, a portion having a pixel value smaller than a predetermined value in the infrared thermography image may be set as the sky. In the embodiment in which the camera is fixed on the road side, the road surface area and the sky area are extracted and stored by the above method once in the daytime on a clear day, and the road surface area and the sky area are thereafter stored based on the stored area. The sky region may be determined. The pixel value is a value indicating the brightness of the pixel in a gray scale image, each value of the RGB components of the pixel (R pixel value, G pixel value, B pixel value) in a color image, or these values. It is possible to adopt a value calculated for and the like.

  The ultraviolet reflectance calculation unit 104 corresponds to the ultraviolet feature amount calculation unit 2 in FIG. The ultraviolet reflectance calculation unit 104 calculates, for each pixel included in the road surface area extracted by the road surface / sky region extraction unit 103, an ultraviolet reflectance at a point on the road surface corresponding to the pixel. Specifically, first, an average value of pixels included in the sky region of the ultraviolet image (referred to as “sky pixel value”) is calculated. Then, an ultraviolet reflectance corresponding to each pixel in the road surface area is calculated based on a ratio obtained by dividing the pixel value of each pixel included in the road surface area of the ultraviolet image by the sky pixel value. For calculation of the sky pixel value, a mode value or a median value may be used instead of the average value. The calculated ultraviolet reflectance is used by the road surface state determination unit 114.

  As described above, ultraviolet rays are likely to be scattered when hitting a substance, and most of the ultraviolet rays are scattered particularly on the snow surface, so that the state of snow can be detected from the ultraviolet ray information. However, if it is determined that snow is present on the road surface if the ultraviolet intensity is equal to or greater than a predetermined threshold value, an erroneous determination may be made due to the influence of variations in the amount of solar radiation itself depending on the season and time zone. For example, if there is a large amount of solar radiation, the UV intensity may exceed the threshold even if there is no snow, and it may be erroneously determined that snow exists. If a large threshold is set to prevent this, conversely, although snow exists and most of the ultraviolet rays are scattered on the snow surface, there is snow because the amount of solar radiation itself is small. It may be misjudged that it is not. Therefore, when it is attempted to detect the snow condition with the ultraviolet intensity, it is necessary to provide a judgment standard according to the season or time zone (for example, judgment by switching a plurality of thresholds depending on the season or time zone).

  Therefore, the ultraviolet reflectance calculation unit 104 adopts the ultraviolet reflectance as a feature quantity that is less susceptible to disturbance than the ultraviolet intensity, and as described above, the pixel value and sky pixel value of each pixel included in the road surface area The ultraviolet reflectance is calculated based on the ratio. Thereby, the influence of the fluctuation | variation of the solar radiation amount of the ultraviolet rays by a season and a time zone is suppressed. According to this method, it is not necessary to change the threshold value for determining the road surface state according to the season or time zone.

  As a cause of disturbance, there is a difference between shade and sun as well as fluctuations in the amount of solar radiation itself, but this is not considered in the ultraviolet reflectance calculation unit 104 in the present embodiment. Originally, ultraviolet rays are easily scattered in the atmosphere, so they are more contained in the sky than the sun's direct light. Due to the nature of being easily scattered, for example, the difference in intensity between the shade of sunlight and the sun is small compared to the difference between the shade of sunlight and the sun, and it is relatively stable regardless of whether it is shade or sun. This is because the state and distribution can be detected.

  The visible light camera imaging unit 106 images the same road surface and sky as captured by the ultraviolet camera imaging unit 101, and the visible light image generation unit 107 generates a visible light image as digital data. The visible light image is not essential for carrying out the present invention, but in the present embodiment, it is necessary for the calculation of the infrared absorption rate described later and the output of the road surface determination result. The visible light camera imaging unit 106 and the visible light image generation unit 107 can use a configuration known as a CCD camera or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, for example. A road surface area is extracted from the generated visible light image by the first road surface area extraction unit 108. As the road surface area extraction method, various methods can be used as in the road surface / sky area extraction unit 103.

  The infrared camera imaging unit 109 captures the same road surface as that captured by the ultraviolet camera imaging unit 101 or the visible light camera imaging unit 106, and an infrared image generation unit 110 generates an infrared image as digital data. For example, the infrared camera imaging unit 109 and the infrared image generation unit 110 can use configurations known as infrared digital CCD cameras. A road surface area is extracted from the generated infrared image by the second road surface area extraction unit 111. As the road surface area extraction method, various methods can be used as in the road surface / sky area extraction unit 103. Further, depending on the embodiment, the first road surface area extraction unit 108 and the second road surface area extraction unit 111 may be common.

  The infrared absorptance calculating unit 112 corresponds to the infrared feature amount calculating means 3 in FIG. The infrared absorptance calculating unit 112 calculates the infrared absorptivity of each pixel included in the road surface area extracted by the second road surface area extracting unit 111 at a point on the road surface corresponding to the pixel. Specifically, for each pixel included in the road surface area of the infrared image, an infrared ray corresponding to each pixel is obtained by calculating a ratio between the pixel value and the pixel value of the pixel on the visible light image corresponding to the pixel. Calculate the absorption rate. The calculated infrared absorption rate is used by the road surface state determination unit 114.

  As described above, infrared rays are easily absorbed by water, and water content meters that actually use infrared rays have been developed. Even in the present invention, the state of water on the road surface is detected using this characteristic of infrared rays. The wavelength region of the infrared light referred to at this time may be limited to about 0.9 μm which is the water absorption wavelength region.

  The reason why the infrared absorptance calculating unit 112 uses the ratio as described above is to suppress the influence of disturbance whether it is shade or sunny. When detecting the state of water, for example, using infrared intensity, and if the infrared intensity is small, it is determined that water is present (determining that the infrared intensity has decreased as a result of the absorption of infrared rays in water) In fact, even if the road surface is dry, it may be erroneously determined that there is water. This is because the amount of infrared radiation contained in the solar radiation is reduced because the amount of solar radiation is small in the shaded area on the road surface. In order to avoid such a misjudgment, in the infrared absorptance calculation unit 112, both the amount of visible light and the amount of infrared light are reduced in the shaded part, so the ratio of the amount of visible light and the amount of infrared light does not change much between the shaded part and the sunny part. I use that. That is, if there is no water on the road surface, the ratio between the pixel value of the visible light image and the pixel value of the infrared image is almost constant regardless of whether it is shaded or sunny. The ratio of the water is absorbed and the water on the road surface can be detected from the degree of the change. Further, by using the ratio, the influence of fluctuations in the amount of solar radiation depending on the season and time zone can be suppressed.

  By the way, in the present invention, since the road surface state is determined using an ultraviolet image and an infrared image, it is necessary that a sufficient amount of ultraviolet rays and infrared rays are applied to the road surface. Therefore, when there is sufficient solar radiation (such as daylight on a clear day), the reflected light reflected by the road surface is used, and when it is not (nighttime or rainy weather), the road surface is illuminated with a lighting device such as a projector. The reflected light of the light is used. Therefore, in the first embodiment, the light projection determination unit 105 determines that there is no sufficient solar radiation when the ultraviolet intensity of the sky region extracted by the road surface / sky region extraction unit 103 is smaller than a predetermined threshold value. The road surface is irradiated with light, and when the intensity of ultraviolet rays in the sky region is equal to or higher than a predetermined threshold, it is determined that there is sufficient solar radiation and the irradiation is stopped. As the ultraviolet intensity in the sky region, for example, the sky pixel value obtained by the ultraviolet reflectance calculation unit 104 can be used. Further, depending on the embodiment, the light projection determination unit 105 not only determines whether to irradiate light, but also adjusts the amount of light when irradiating light from the illumination device according to the ultraviolet intensity in the sky region. May be performed. For example, the amount of light emitted from the illumination device may be reduced as the ultraviolet intensity in the sky region increases, and adjustment may be performed so that light is not emitted from the illumination device when the ultraviolet intensity in the sky region is equal to or greater than a predetermined threshold.

  In the conventional road surface condition determination by visible light image, there is a problem that misjudgment is likely to occur at the boundary between day and night. In this way, by switching between sunlight and light from the lighting device, it is highly accurate throughout the day and night. It is possible to determine the road surface condition. Note that a lighting device that can irradiate light including ultraviolet rays and infrared rays is used. The illumination device may use a device having a relatively uniform spectral distribution such as a xenon lamp, or may use sunlight illumination having a spectral distribution similar to sunlight. Existing street lamps on the roadside or vehicle headlamps may be used. If the spectral distribution of the light emitted from the lighting device is known in advance, the calculated values in the ultraviolet reflectance calculation unit 104 and the infrared absorption rate calculation unit 112 may be corrected based on the spectral distribution.

  The infrared thermography imaging unit 113 also images the road surface captured by the ultraviolet camera imaging unit 101, the visible light camera imaging unit 106, and the infrared camera imaging unit 109, and generates a temperature distribution image as digital data by the temperature distribution image generation unit 117. . The infrared thermography imaging unit 113 and the temperature distribution image generation unit 117 can use a known configuration as an infrared thermography device. The temperature distribution image represents the road surface temperature and is used by the road surface state determination unit 114.

  Infrared rays are always emitted from objects that have heat, and the amount of infrared radiation and temperature have a correlation. An infrared thermography apparatus is a camera that measures the temperature of an imaged area using this property.

  The road surface state determination unit 114 corresponds to the road surface state determination unit 4 in FIG. The road surface state determination unit 114 receives the ultraviolet reflectance, infrared absorption rate, and temperature distribution image from the ultraviolet reflectance calculation unit 104, the infrared absorption rate calculation unit 112, and the temperature distribution image generation unit 117, and based on these inputs. Determine the road surface condition. The determination in the present embodiment corresponds to classification into one of predetermined road surface condition classification items. For example, the classification item “pressed snow, compressed snow ice plate, sherbet, pomegranate, dry, wet, ice film, water film, ice plate” is determined in advance, and the road surface condition determination unit 114 corresponds to any of these classification items. Determine. Since the classification items can be arbitrarily determined, it is preferable that the classification items are appropriately subdivided and determined so as to be easily understood by a person viewing the determination result, and in accordance with the local climate in which the road surface condition is determined.

  Although various methods can be used as the determination method, in the first embodiment, the road surface state is determined according to a predetermined determination condition. For example, FIG. 3 (a) shows the approximate value of the ultraviolet reflectance obtained from the experiment for each road surface condition. Based on this ultraviolet reflectance, the case where the ultraviolet reflectance is 80% or more is defined as snow. You may judge. In this embodiment, it is determined that the snow pressure and the sherbet are in descending order of the ultraviolet reflectance, and when the ultraviolet reflectance is low, the water film, the wet, and the dry are determined in the descending order of the infrared absorption rate. (B) of FIG. 3 shows the determination conditions. When threshold values A, B, C, and D are used (A <B, C <D), and the ultraviolet reflectance is D or more, C or more and D If it is less than C, the case is less than C. Also, the distinction between dry, wet, and water film is difficult to judge from the ultraviolet reflectance, so that the ultraviolet reflectance is less than C, and further, when the infrared absorption rate is B or more, when A or more and less than B, A It is divided into the cases of less than. And in order to determine a frozen state about each classification | category, it divides into the case where road surface temperature is higher than 0 degreeC, and the case where it is below 0 degreeC. The values of A, B, C, and D are determined in advance by obtaining appropriate values through experiments or the like (for example, D = 80% is set from the ultraviolet reflectance in FIG. 3A). In FIG. 3B, the case where the road surface temperature is higher than 0 ° C. is shown to the left of “/”, and the case where the road surface temperature is 0 ° C. or less is shown to the right of “/”. For example, when the ultraviolet reflectance is C or more and less than D, regardless of the infrared absorptivity, if the road surface temperature is higher than 0 ° C., it is determined as “Sherbet” (a state in which snow melts and becomes mixed with water). If the temperature is 0 ° C. or lower, it is determined as “creep” (crisp ice state). Since ice has a correlation with the road surface temperature, for example, when the ultraviolet reflectance is less than C, it is known from the distribution of water detected from the value of the infrared absorption rate and the temperature distribution image generated by the temperature distribution image generation unit 117. From the road surface temperature, the ice condition (ice film, ice plate, etc.) can be determined.

  The above threshold values A, B, C, D and the classification item data corresponding to each area (b) in FIG. 3 (such as “compressed snow”) are stored in the road surface state determination unit 114 or other components. Is stored in advance in the storage means shared with. The step of obtaining the road surface state classification item from the input ultraviolet reflectance, infrared absorption rate, and temperature distribution image may be realized in software by the program stored in the storage means, and the road surface state determination unit 114. It may be realized by providing a dedicated hardware circuit. The storage means may be a nonvolatile memory such as a ROM (Read Only Memory). Alternatively, the storage means may be a RAM (Random Access Memory). In this case, a portable storage medium (flexible disk, CD (Compact Disc), DVD (Digital Versatile Disk), optical disk, or the like in which necessary data and programs are stored in advance. The stored contents are read in advance from a magnetic disk or the like into the RAM, or necessary data or programs are read in advance from the network through a communication interface (not shown).

  The road surface state determination unit 114 may perform the above determination for each pixel, or may divide the road surface into a plurality of regions and perform the above determination for each region. For example, the road surface is divided so that each area has vertical N pixels × horizontal M pixels, and an average value of N × M pixel values is obtained for each area, and the average value is used to determine the area of FIG. You may determine according to determination conditions. Another value such as a mode value may be used instead of the average value.

The road surface state image generation unit 115 generates a road surface state image based on the road surface state determined by the road surface state determination unit 114 and outputs the road surface state image to the road surface determination result output unit 116.
The road surface state image is, for example, an image in which the road surface is divided into a plurality of regions and the road surface state is color-coded for each region. For example, dry is displayed in blue, wet or water film is displayed in yellow, sherbet or compressed snow is displayed in orange, compressed snow ice plate, pomegranate, ice film, or ice plate is displayed in red. The display color is determined in advance according to the road surface condition so that a human can easily grasp the dangerous location and the degree of the danger. Color coding may be performed not for each region but for each pixel.

  Further, since the freezing event is an estimation from the road surface temperature, it may be expressed separately from other events. For example, the state of snow or water determined from ultraviolet rays or infrared rays may be expressed by color, and the degree of freezing determined from road surface temperature may be expressed by means other than color. Specifically, for example, a region where the road surface temperature is 0 ° C. or lower is expressed so as to appear three-dimensionally high, and a region where the road surface temperature is higher than 0 ° C. is expressed so as not to have a height (looks flat). In this way, if the road surface is expressed as an image with unevenness, it becomes easier for the driver to intuitively grasp the road surface risk level. In addition, by combining the expression of color and height, it is possible to represent the combined state of the road surface state related to snow and water and the road surface state related to freezing in one image, so that humans can easily grasp the dangerous state of the road surface. be able to. Or you may use only the expression by height, such as expressing high as the road surface condition with high risk.

  The road surface determination result output unit 116 combines the road surface state image with the visible light image generated by the visible light image generation unit 107 (which is a color image in this embodiment, but may be a gray scale image), and the road surface state determination result. Output an image. An example of the road surface state determination result image is shown in FIG. FIG. 4 is an example of a road surface state determination result image in which a road surface state image obtained by dividing a road surface into a plurality of regions and color-coding the road surface state for each region is combined with a visible light image. In FIG. 4, the pressure snow is displayed in yellow and the freeze is displayed in red, so that it is easy to grasp the dangerous state of the road surface.

  In this embodiment, it is assumed that the difference in viewpoint position among the ultraviolet camera imaging unit 101, the visible light camera imaging unit 106, the infrared camera imaging unit 109, and the infrared thermography imaging unit 113 is negligible. That is, if the images captured by these imaging units and generated by the ultraviolet image generation unit 102, the visible light image generation unit 107, the infrared image generation unit 110, and the temperature distribution image generation unit 117 have the same size. It can be assumed that the same point on the road surface corresponds to a pixel having the same coordinate in each image, and that no coordinate conversion processing is required. Depending on the embodiment, the difference in the viewpoint position of each imaging unit may not be negligible, but in that case, as described later, it corresponds to which coordinate of each image the same one point on the road surface corresponds to It is sufficient to further provide means for taking

  FIG. 5 is a functional block diagram of a road surface condition determination apparatus according to the second embodiment of the present invention. In the second embodiment, an ultraviolet image, an infrared image, and a visible light image are captured using one stereo color camera. A road surface state can be determined from an ultraviolet image and an infrared image, and a visible light image can be used for applications such as outputting a road surface determination result. Also in this embodiment, the ultraviolet reflectance is used as the ultraviolet feature quantity, and the infrared absorption rate is used as the infrared feature quantity. Although the visible light image is not an essential element for the implementation of the present invention, in the present embodiment, the second method that is preferable as the above-described method for calculating the infrared feature amount is adopted, so the visible light image is also captured and used. .

  The road surface condition determination apparatus according to the second embodiment includes a blue wavelength region cut unit 301, a first imaging unit 302, a first image generation unit 303, a road surface / sky region extraction unit 304, an ultraviolet reflectance calculation unit 305, a projection unit. Light determination unit 306, red wavelength region cut unit 307, second imaging unit 308, second image generation unit 309, road surface region extraction unit 310, infrared absorption rate calculation unit 311, relative position calculation unit 312, infrared thermography imaging unit 313, a road surface state determination unit 314, a road surface state image generation unit 315, a color image synthesis unit 316, a road surface determination result output unit 317, and a temperature distribution image generation unit 318.

  The first imaging unit 302 and the second imaging unit 308 are two imaging units of a stereo color camera. The first imaging unit 302 is sensitive to at least the ultraviolet region and the visible light region (including all of the red wavelength region, the green wavelength region, and the blue wavelength region), and the second imaging unit 308 is at least the infrared region and the visible light region. It has sensitivity in the region (including all of the red wavelength region, green wavelength region, and blue wavelength region). Both the first imaging unit 302 and the second imaging unit 308 may be imaging units using light receiving elements having sensitivity in the wavelength region from the ultraviolet region to the infrared region.

  For example, some common visible light image capturing cameras (CCD color cameras, CMOS color cameras, etc.) use light receiving elements having sensitivity in the wavelength region from the near ultraviolet region to the near infrared region. is there. In such a camera, the spectral sensitivity characteristics of the light receiving element are different from those of the human eye, so that the visible light image captured by the human eye is close to the image seen by the human eye after the ultraviolet rays and infrared rays are cut off by the filter. I am doing so. In the present embodiment, for example, a filter that cuts off ultraviolet rays and infrared rays can be removed from a stereo color camera including two such cameras, and the first imaging unit 302 and the second imaging unit 308 can be used. .

  By the way, in general, in a camera using a solid-state imaging device such as a CCD or a CMOS, the light receiving device itself does not have the ability to distinguish colors. Therefore, the light is red (R) • using a RGB color filter or a spectral prism. Color imaging is performed by decomposing the light into green (G) and blue (B) and separately receiving light by the light receiving element, and obtaining a color image by obtaining RGB pixel values. In a typical CCD camera or CMOS camera for capturing a visible light image, color separation is performed on light (that is, visible light) after passing through the above-described filter that cuts off ultraviolet rays and infrared rays. Therefore, the color filter and the spectral prism used for color separation need not select only visible light red, green, and blue. In other words, even if you use a color filter or spectral prism that does not separate light in the red wavelength region of visible light from light in the longer wavelength region (that is, infrared light), the infrared light is already cut by the filter. Alternatively, only the light in the red wavelength region of visible light is extracted by the spectral prism, and the R pixel value is obtained. Similarly, even if a color filter or a spectral prism that does not distinguish between light in the blue wavelength region of visible light and light in the shorter wavelength region (that is, ultraviolet light) is used, this color is already cut off by the filter. Only the light in the blue wavelength region of visible light is extracted by the filter or the spectral prism, and the B pixel value is obtained. However, only the G pixel value corresponding to light in the green wavelength region of visible light needs to be extracted using a color filter or a spectral prism.

  In this embodiment, the color filter and the spectral prism are included in the first imaging unit 302 and the second imaging unit 308 as components. When described as using a color filter, in this embodiment, the blue (B) color filter is a filter that transmits light in the blue wavelength region of visible light and light in a shorter wavelength region (that is, ultraviolet light), green (G). The color filter is a filter that transmits light in the green wavelength region of visible light, and the red (R) color filter is a filter that transmits light in the red wavelength region of visible light and light in a longer wavelength region (that is, infrared light).

  A blue wavelength region cut unit 301 and a red wavelength region cut unit 307 are provided in front of the first imaging unit 302 and the second imaging unit 308, respectively. The blue wavelength region cut unit 301 is a filter that cuts light in the blue wavelength region of visible light, and corresponds to the filter means in FIG. The red wavelength region cut unit 307 is a filter that cuts light in the red wavelength region of visible light, and corresponds to the filter unit in FIG. The functions of the blue wavelength region cut unit 301 and the red wavelength region cut unit 307 and FIG. 6 will be described later together with the functions of the first image generation unit 303 and the second image generation unit 309. In addition, when using what removed the filter which cuts an ultraviolet-ray and infrared rays from the CCD camera and CMOS camera for general visible light images as mentioned above as the 1st image pick-up part 302 and the 2nd image pick-up part 308, Instead of the removed filter, a blue wavelength region cut unit 301 and a red wavelength region cut unit 307 are provided in front of the imaging unit.

  As in the first embodiment, the first imaging unit 302 captures the road surface and the sky together, and the first image generation unit 303 generates an image as digital data. In the following processing, since calculation is performed using the RGB components of the image generated here, the first image generation unit 303 is not a full-color image but an image decomposed into RGB elements (hereinafter, referred to as “color image”). In the following description, it is assumed that “R image”, “G image”, and “B image” are respectively generated. However, depending on the embodiment, the first image generation unit 303 generates a full-color image, and the ultraviolet reflectance calculation unit 305 or the like obtains pixel values of necessary elements among the RGB elements from the image data. It is good.

The B image generated by the first image generation unit 303 corresponds to an ultraviolet image. The reason is as follows.
FIG. 6 is a diagram showing the principle of generating an infrared image and an ultraviolet image using a stereo color camera. Among these, (b) shows that the B image generated by the first image generating unit 303 is an ultraviolet image. Is explained. The filter means in the figure corresponds to the blue wavelength region cut unit 301, where the blue wavelength region is a blocking region and the ultraviolet region is a transmission region. The B image is generated based on the light that has passed through the blue color filter. As described above, the blue color filter is light in the blue wavelength region of visible light and light in a shorter wavelength region (that is, ultraviolet light). Permeate. Therefore, it is ultraviolet light that passes through the blue wavelength region cut unit 301 and is received by the first imaging unit 302 through the blue color filter, so that the B image generated by the first image generation unit 303 is received. Corresponds to an ultraviolet image.

  The G image generated by the first image generation unit 303 is an image corresponding to light in the green wavelength region. The R image generated by the first image generation unit 303 becomes an image based on the light in the red wavelength region and the infrared ray if the first imaging unit 302 has sensitivity to infrared rays. If is not sensitive to infrared light, an image based on light in the red wavelength region is obtained.

  From the B image generated by the first image generation unit 303, the road surface area and the celestial sphere area are extracted by the road surface / sky area extraction unit 304. The road surface / sky region extraction unit 304 uses the same road surface region and sky region extraction method as the road surface / sky region extraction unit 103 in the first embodiment.

  The ultraviolet reflectance calculation unit 305 corresponds to the ultraviolet feature amount calculation unit 2 in FIG. The ultraviolet reflectance calculation unit 305 calculates, for each pixel included in the road surface area extracted by the road surface / sky region extraction unit 304, the ultraviolet reflectance at a point on the road surface corresponding to the pixel. The calculation method is the same as that of the ultraviolet reflectance calculation unit 104 in the first embodiment. The calculated ultraviolet reflectance is used by the road surface state determination unit 314.

  The second embodiment also includes a light projection determination unit 306 similar to that of the first embodiment, and determines whether or not to irradiate light from the lighting device according to the amount of solar radiation. The lighting device is controlled according to this determination result.

  The operation principle of the red wavelength region cut unit 307, the second imaging unit 308, and the second image generation unit 309 is shown in FIG. 6A, and the blue wavelength region cut shown in FIG. The operation principles of the unit 301, the first imaging unit 302, and the first image generation unit 303 are the same. The fact that the operating principle is the same is also clear from the fact that (a) and (b) in FIG. 6 are symmetrical. In either case, an imaging unit (second imaging unit 308 or first imaging unit 302) having sensitivity to a first wavelength region (infrared ray or ultraviolet ray) to be imaged and a second wavelength region (visible light) adjacent thereto. The filter means (red wavelength region) using the third wavelength region that is the wavelength region closest to the first wavelength region among the red, green, and blue wavelength regions included in the second wavelength region as a cutoff region The desired infrared image or ultraviolet image is obtained by separating the light transmitted through the cut unit 307 or the blue wavelength region cut unit 301) into three colors of RGB.

  That is, since the red wavelength region cut unit 307 cuts light in the red wavelength region of visible light, the R image generated by the second image generation unit 309 corresponds to an infrared image. The G image generated by the second image generation unit 309 is an image corresponding to light in the green wavelength region. The B image generated by the second image generation unit 309 is an image based on the light in the blue wavelength region and the ultraviolet ray if the second imaging unit 308 is sensitive to ultraviolet rays. If is not sensitive to ultraviolet light, an image based on light in the blue wavelength region is obtained.

  By the way, since the color that the human eye feels most bright is green, there is no significant problem even if a G image is used instead of a full-color visible light image as a visible light image used for calculation in the process of determining the road surface condition. Conceivable. Therefore, in the second embodiment, an R image generated by the second image generation unit 309 is provided as an infrared image to the road surface region extraction unit 310, and is generated by the second image generation unit 309 as a visible light image. Give a G image. Then, the R and G images from which the road surface area has been extracted by the road surface area extraction unit 310 are provided to the infrared absorptance calculation unit 311. The method for extracting the road surface area and the method for calculating the infrared absorptance from the two images are the same as in the first embodiment. The infrared absorptance calculating unit 311 corresponds to the infrared feature amount calculating unit 3 in FIG.

  The relative position calculation unit 312 compares the two G images generated by the first image generation unit 303 and the second image generation unit 309, and two images captured by the stereo color camera (two imaging units). Relative position information is calculated. Although the calculation method is well known, it is omitted. For example, the relative position information is calculated by extracting the contour lines of two G images and calculating the deviation between the X and Y axes of the two images by template matching.

  The infrared thermography imaging unit 313 and the temperature distribution image generation unit 318 are the same as the infrared thermography imaging unit 113 and the temperature distribution image generation unit 117 in the first embodiment.

  The road surface state determination unit 314 corresponds to the road surface state determination unit 4 in FIG. The road surface state determination unit 314 receives the ultraviolet reflectance, the infrared absorption rate, and the temperature distribution image from the ultraviolet reflectance calculation unit 305, the infrared absorption rate calculation unit 311, and the temperature distribution image generation unit 318, and based on these inputs. Determine the road surface condition. In addition, since the stereo color camera is originally used for stereoscopic image recognition and the like, the interval between the two imaging units is set to such an extent that the difference between the viewpoints of the two imaging units cannot be ignored. Since the ultraviolet reflectance and the infrared absorptance are calculated based on images picked up by these two different image pickup units, the same point on the road surface corresponds to a pixel having a different coordinate in each image. ing. Therefore, in order to determine the road surface state at the location from the ultraviolet reflectance and infrared absorption rate at a location, it is necessary to correspond the coordinates of the two images. The relative position information calculated by the relative position calculation unit 312 is information corresponding to the difference between the viewpoints of the two imaging units. Therefore, the relative position information calculated by the relative position calculation unit 312 is given to the road surface state determination unit 314.

  The road surface state determination unit 314 may perform determination by the same method as in the first embodiment, but in the second embodiment, the road surface state determination unit 314 is configured by a neural network. A road surface state determination method using a neural network will be described with reference to FIGS.

  FIG. 7 is a diagram illustrating an example of a pixel arrangement configuration of an image. Below, the road surface state determination part 314 shall determine a road surface state with respect to each pixel of each pixel, and demonstrates the example in the case of determining the road surface state of the pixel (3, 2) filled. In the following, it is assumed that the correspondence between the coordinates of the two images has already been taken based on the relative position information. For example, the coordinates of the image used for calculating the infrared absorptance are converted according to the relative position information and matched with the coordinates used for the calculation of the ultraviolet reflectance, so that the pixels having the same coordinates in both images are set to the same point on the road surface. In order to correspond, it demonstrates that it is after finishing a process. As will be described later, in order to determine the road surface state of the pixel (3, 2), eight pixels surrounded by a thick line in the vicinity thereof, that is, pixels in the range of (2, 1) to (4, 3) Pixel values are also used.

  FIG. 8 is a diagram showing an example in which a neural network is configured for the pixel (3, 2). Since the road surface state determination unit 314 determines the road surface state for each pixel of each pixel, a neural network similar to FIG. 8 is configured for each pixel. The neural network has a hierarchical structure including an input layer 501, an intermediate layer 502, and an output layer 503. Although the intermediate layer 502 is one layer in FIG. 8, it may be a multilayer.

  In the operation process of the neural network, the ultraviolet reflectances of (3, 2) which is the target pixel of the road surface condition determination and eight pixels in the range of (2, 1) to (4, 3) in the vicinity are The infrared absorption rate is input to the input layer 501. Numerical values corresponding to the road surface condition through the intermediate layer 502 (for example, 0.1 for dry, 0.3 for wet, 0.5 for water film, 0.7 for sherbet, 0.9 for compressed snow, etc.) ) Is output. By inputting not only the target pixel but also the ultraviolet reflectance and infrared absorption rate of neighboring pixels, the output accuracy can be further increased. Note that the range of pixels to be selected as neighboring pixels may be arbitrarily determined depending on the embodiment. The road surface state is determined based on the output value from the output layer 503 and the temperature obtained from the temperature distribution image. For example, when the output from the output layer 503 is a value representing wetness and the road surface temperature is higher than 0 ° C., it is determined to be wet, and when the road surface temperature is 0 ° C. or lower, it is determined to be an ice film (see (b) of FIG. 3). (Refer to the “wet / ice film” pair in the judgment condition).

  By the way, in order to operate the neural network, it is necessary to first learn weights (weights) at the nodes of the intermediate layer 502 and the output layer 503. In the learning process, the weight change amount is calculated by providing the neural network with the ultraviolet reflectance and infrared absorption rate of the target pixel (3, 2) and neighboring pixels as input data, and giving numerical values according to the road surface condition as teacher data. To do. By repeating this, an optimum weight can be calculated. A numerical value corresponding to the road surface condition may be given by a human, or a numerical value calculated from an output value of a meteorological instrument such as a rain gauge or a depth gauge may be automatically given. The criteria for determining the road surface condition visually by humans differ from person to person, but if the neural network is trained by providing teacher data according to each person, the road surface condition can be determined according to the reference according to each person. become able to.

  Similarly to the road surface state image generation unit 115 in the first embodiment, the road surface state image generation unit 315 generates a road surface state image based on the road surface state determined by the road surface state determination unit 314 and outputs a road surface determination result. To the unit 317.

  The color image composition unit 316 receives the images generated by the first image generation unit 303 and the second image generation unit 309 and the relative position information calculated by the relative position calculation unit 312 as inputs, and combines the color images. And output to the road surface determination result output unit 317. This color image corresponds to a visible light image given as an input to the road surface determination result output unit 116 in the first embodiment.

  For example, when synthesizing an image having the first imaging unit 302 as a viewpoint, the color image is synthesized as follows. First, the B image generated by the second image generation unit 309 is converted into a B image having the first imaging unit 302 as a viewpoint based on the relative position information. When the second imaging unit 308 is sensitive to ultraviolet rays, the B image is an image based on light in the blue wavelength region and ultraviolet rays, and therefore, the B image (ultraviolet image) generated by the first image generation unit 303 A difference image is obtained, and the difference image is set as a B image corresponding to light in the blue wavelength region. As the G image corresponding to the light in the green wavelength region, the G image generated by the first image generation unit 303 is used. The R image corresponding to the light in the red wavelength region is also obtained by a procedure similar to that for the B image. That is, the R image (infrared image) generated by the second image generation unit 309 is converted into an R image having the first imaging unit 302 as a viewpoint based on the relative position information. When the first imaging unit 302 has sensitivity to infrared rays, the R image generated by the first image generation unit 303 is an image based on light in the red wavelength region and infrared rays. And the difference image is set as an R image corresponding to light in the red wavelength region. A color image is synthesized from the R image, G image, and B image thus obtained.

  Depending on the embodiment, an image with the second imaging unit 308 as a viewpoint may be synthesized, and an image with an intermediate point between the first imaging unit 302 and the second imaging unit 308 as a viewpoint is synthesized. You may do it. Further, without obtaining the difference image, for example, the R image generated by the first image generation unit 303 may be used as it is as the R image for synthesizing the color image. The images required for color image synthesis differ depending on the embodiment, but the color image synthesis unit 316 receives the necessary images from the first image generation unit 303 and the second image generation unit 309 as inputs. Shall.

  Similarly to the road surface determination result output unit 116 in the first embodiment, the road surface determination result output unit 317 combines the color image combined by the color image combining unit 316 and the road surface state image generated by the road surface state image generation unit 315. The determination result image is output.

  In addition, there is a system that uses an in-vehicle stereo camera for obstacle detection, etc., but by sharing the camera and the captured image with such an existing system, the road surface state determination device according to the second embodiment Can be realized at low cost.

  FIG. 9 is a functional block diagram of a road surface condition judging device according to the third embodiment of the present invention. In the third embodiment, an ultraviolet image, a visible light image, and an infrared image are captured by a single imaging device (general-purpose color camera). A road surface state can be determined from an ultraviolet image and an infrared image, and a visible light image can be used for applications such as outputting a road surface determination result. Also in this embodiment, the ultraviolet reflectance is used as the ultraviolet feature quantity, and the infrared absorption rate is used as the infrared feature quantity. The visible light image is not an essential element for carrying out the present invention. However, in the present embodiment, the second method that is preferable as the method for calculating the above-described ultraviolet feature amount is adopted.

  The road surface state determination apparatus according to the third embodiment includes a blue wavelength region / red wavelength region cut unit 401, an imaging unit 402, an image generation unit 403, a road surface region extraction unit 404, an ultraviolet reflectance calculation unit 405, a road surface / sky region extraction. Unit 406, light projection determination unit 407, infrared absorption rate calculation unit 408, infrared image imaging unit 409, temperature measurement unit 410, temperature distribution image generation unit 411, road surface state determination unit 412, road surface state image generation unit 413, road surface determination result An output unit 414.

  The blue wavelength region / red wavelength region cut unit 401 cuts the light in the blue wavelength region and the red wavelength region in the visible light, and passes the ultraviolet light, the light in the green wavelength region in the visible light, and the infrared light. It is a filter. As shown in FIG. 10, the blue wavelength region / red wavelength region cut unit 401 is a filter that functions as both the blue wavelength region cut unit 301 and the red wavelength region cut unit 307 (FIG. 6) in the second embodiment. It can also be regarded as a means. As apparent from the comparison between FIG. 6 and FIG. 10, depending on the embodiment, the blue wavelength region / red wavelength region cut unit 401 is not realized by a triple band pass filter, but the blue wavelength in the second embodiment. It can also be realized by configuring with two filters similar to the region cut unit 301 and the red wavelength region cut unit 307. That is, by configuring the filter in two stages, both the light in the blue wavelength region and the light in the red wavelength region may be cut by the respective filters, and the same effect as the triple band pass filter may be obtained as a whole. .

  The imaging unit 402 receives and captures the light that has passed through the blue wavelength region / red wavelength region cut unit 401. An imaging unit 402 having sensitivity from ultraviolet rays to infrared rays is used. As described above, many CCD color cameras and CMOS color cameras generally used for visible light images use light receiving elements having sensitivity from near ultraviolet rays to near infrared rays. Camera parts can be used for the imaging unit 402. Similar to the first imaging unit 302 and the second imaging unit 308 in the second embodiment, the imaging unit 402 includes a color filter, a spectral prism, and the like as its constituent elements in addition to the imaging element. When using a CCD color camera or a CMOS color camera component for visible light image as the imaging unit 402, the filter for cutting off ultraviolet rays and infrared rays is removed and used as in the second embodiment.

  Similar to the first and second embodiments, the imaging unit 402 captures the road surface and the sky together, and the image generation unit 403 generates an image as digital data. In the following description, it is assumed that the image generation unit 403 generates an R image, a G image, and a B image as in the second embodiment. However, a full color image may be generated (this also applies to the second embodiment). It is the same as the form).

  The R image generated by the image generation unit 403 corresponds to an infrared image, the G image corresponds to a visible light image, and the B image corresponds to an ultraviolet image. The reason is the same as described in the second embodiment, and is clear from FIG. In this way, in the present embodiment, an infrared image, a visible light image, and an ultraviolet image are obtained from one imaging unit 402.

  The B image and G image generated by the image generation unit 403 are input to the road surface region extraction unit 404, and the road surface regions are extracted by the method described above, and output to the ultraviolet reflectance calculation unit 405.

  The ultraviolet reflectance calculation unit 405 corresponds to the ultraviolet feature amount calculation unit 2 in FIG. The ultraviolet reflectance calculation unit 405 calculates, for each pixel included in the road surface area extracted by the road surface area extraction unit 404, the ultraviolet reflectance at a point on the road surface corresponding to the pixel. Specifically, for each pixel included in the road area of the B image, the ultraviolet reflectance corresponding to each pixel is based on the ratio of the pixel value divided by the pixel value of the pixel on the G image corresponding to that pixel. Is calculated. That is, the ultraviolet reflectance is calculated based on the ratio between the intensity of visible light and ultraviolet light. If there is snow on the road surface, a lot of ultraviolet rays are scattered on the snow surface and the intensity of ultraviolet rays increases. On the other hand, the intensity of visible light does not change so much, so the value of this ratio also increases. Therefore, the ultraviolet reflectance is calculated based on this ratio. By the calculation method based on the ratio, the influence of fluctuations in solar radiation can be suppressed. The calculated ultraviolet reflectance is used by the road surface state determination unit 412.

  Note that ultraviolet rays are more easily scattered than visible rays and infrared rays, and the difference in intensity between the shade and the sun is relatively small. Therefore, the influence of the difference between the shade and the sun cannot be completely eliminated by the above method. However, comparing the shade and the sun, the intensity of both ultraviolet rays and visible light is low in the shade, and the intensity of both ultraviolet rays and visible light is high in the sun. Therefore, the difference between the shade and the sun is calculated by using the ratio as described above. The influence of disturbance due to can be suppressed to some extent.

  The R image generated by the image generation unit 403 is input to the road surface / sky region extraction unit 406, the road surface region and the sky region are extracted by the method described above, and the infrared absorption rate calculation unit 408 and the light projection determination unit 407 are extracted. Each is output.

  The infrared absorptance calculating unit 408 corresponds to the infrared feature amount calculating means 3 in FIG. The infrared absorption rate calculation unit 408 calculates the infrared absorption rate at a point on the road surface corresponding to the pixel for each pixel included in the road surface region extracted by the road surface / sky region extraction unit 406. Specifically, first, an average value (sky pixel value) of pixels included in the sky region of the R image is calculated, and the pixel value of each pixel included in the road surface region of the R image is divided by the sky pixel value. Based on the ratio, an infrared absorption rate corresponding to each pixel is calculated. In calculating the sky pixel value, the mode value or the median value may be used instead of the average value. The calculated infrared absorption rate is used by the road surface state determination unit 412.

  If it is going to judge the state regarding the water on a road surface from the intensity | strength of the infrared rays on a road surface, it will be influenced by the fluctuation | variation of the solar radiation amount by a season and a time zone. Therefore, for example, it is necessary to change the threshold value for determining whether it is dry or wet according to the season or time zone. However, if the calculation method based on the ratio as described above is used, the influence of fluctuations in the amount of solar radiation can be eliminated.

  Similarly to the first and second embodiments, the third embodiment also includes a light projection determination unit 407 that performs determination according to the amount of solar radiation. However, in the third embodiment, the determination is based not on the amount of solar radiation but on the amount of infrared radiation. Therefore, the pixel value of the R image extracted by the road surface / sky region extraction unit 406 is used for the determination. For example, the average value of the pixel values of the road surface area is used as a value indicating the amount of infrared radiation, and whether or not to irradiate light from the lighting device depending on whether or not this value is greater than a predetermined threshold value. May be determined. The light projection determination unit 407 may not only determine whether or not to irradiate, but may also adjust the amount of light in the case of irradiation according to a value indicating the amount of solar radiation.

  Based on inputs from the infrared image capturing unit 409 and the temperature measuring unit 410, a temperature distribution image generating unit 411 generates a temperature distribution image. Specifically, the infrared image capturing unit 409 is an infrared camera.

  In general, an infrared thermography device (corresponding to the infrared thermography imaging unit 113 and the temperature distribution image generation unit 117, or the infrared thermography imaging unit 313 and the temperature distribution image generation unit 318) is an infrared thermometer such as a sensor for measuring the environmental temperature. It has an internal mechanism for associating the amount of radiant energy with temperature. Therefore, it is more expensive than a simple infrared camera. Therefore, in the third embodiment, an infrared image capturing unit 409 and a temperature measuring unit 410 are used instead of the infrared thermography device in order to obtain a temperature distribution image using inexpensive means.

  Specifically, the temperature measurement unit 410 is a temperature sensor such as a thermocouple or a radiation thermometer, and is included in the range of the road surface (which is also an imaging target of the imaging unit 402) captured by the infrared image capturing unit 409. Measure the point temperature. First, the temperature distribution image generation unit 411 obtains pixel values corresponding to the two points in the infrared image captured by the infrared image capturing unit 409 (each pixel value of one pixel may be obtained, An average value of pixel values of pixels included in a small area near the pixel may be obtained). Then, based on the two temperatures (t1, t2) and the two pixel values (p1, p2), a value A for converting the pixel value of the infrared image into a temperature is calculated, and based on the value of A, The pixel value of each pixel of the infrared image captured by the infrared image capturing unit 409 is converted to generate a temperature distribution image. Thereby, the temperature distribution of the whole road surface is computable. For example, B and C are constants, A is calculated by A = B × (t1−t2) / (p1−p2), and converted by t = p × A + C based on the pixel value p of the infrared image. A temperature distribution image may be generated from the measured temperature t. The calculation formulas for A and t may be arbitrarily determined according to the embodiment.

  The road surface state determination unit 412 corresponds to the road surface state determination unit 4 in FIG. The road surface condition determination unit 412 receives the ultraviolet reflectance, infrared absorption rate, and temperature distribution image from the ultraviolet reflectance calculation unit 405, the infrared absorption rate calculation unit 408, and the temperature distribution image generation unit 411, and based on these inputs. Determine the road surface condition. It may be determined by the same method as in the first or second embodiment, or may be determined by other methods.

The road surface state image generation unit 413 generates a road surface state image as in the first and second embodiments, and outputs the road surface state image to the road surface determination result output unit 414.
In the present embodiment, the road surface determination result output unit 414 receives the G image generated by the image generation unit 403 as a visible light image, for example, a G image (or an image obtained by converting the G image into a gray scale). The road surface state image input from the road surface state image generation unit 413 is synthesized to be a determination result image and output.

  In addition, by sharing a camera with an existing system such as a vehicle monitoring system on a road, the road surface state determination device according to the third embodiment can be realized at low cost. Of course, not only the camera but also the captured image can be shared with the existing system.

The road surface condition determination apparatus according to the present invention is not limited to the above-described embodiment, and various changes can be made. The main modifications will be specifically described below.
In the first embodiment, when the difference in viewpoint position among the ultraviolet camera imaging unit 101, the visible light camera imaging unit 106, the infrared camera imaging unit 109, and the infrared thermography imaging unit 113 cannot be ignored, in the second embodiment It is desirable to further provide means similar to the relative position calculation unit 312. That is, the relative position information of each image captured by these imaging units and generated by the ultraviolet image generation unit 102, the visible light image generation unit 107, the infrared image generation unit 110, and the temperature distribution image generation unit 117 is calculated on the road surface. It is desirable to provide a means for taking correspondence to which coordinate of each image corresponds to the same point of the image, and to provide the road surface condition determination unit 114 with the relative position information calculated by the means.

  Similarly, also in the second and third embodiments, when the difference in viewpoint position between the infrared thermography imaging unit 313 or the infrared image imaging unit 409 and another imaging unit cannot be ignored, the temperature distribution image generation unit 318 is used. It is desirable to calculate relative position information of the temperature distribution image generated in 411 and other R image, G image, and B image, and to provide the calculated relative position information to the road surface state determination units 314 and 412 as well.

  In the first or second embodiment, the road surface / sky region extraction units 103 and 304 and the ultraviolet reflectance calculation units 104 and 305 are replaced with the road surface region extraction unit 404 and the ultraviolet reflectance calculation in the third embodiment. A configuration similar to that of the unit 405 can be used. Conversely, the road surface area extraction unit 404 and the ultraviolet reflectance calculation unit 405 in the third embodiment can be replaced with the same configuration as in the first or second embodiment.

  Further, the road surface area extraction units 108, 111, 310 and the infrared absorption rate calculation units 112, 311 in the first or second embodiment are used as the road surface / sky region extraction unit 406, infrared absorption in the third embodiment. A configuration similar to that of the rate calculation unit 408 can be used. Conversely, the road surface / sky region extraction unit 406 and the infrared absorption rate calculation unit 408 in the third embodiment can be replaced with the same configuration as in the first or second embodiment.

  Further, the infrared thermography imaging units 113 and 313 and the temperature distribution image generation units 117 and 318 in the first or second embodiment are replaced with the infrared image imaging unit 409, the temperature measurement unit 410, and the temperature in the third embodiment. A configuration similar to that of the distribution image generation unit 411 can be used. Conversely, the infrared image capturing unit 409, the temperature measurement unit 410, and the temperature distribution image generation unit 411 in the third embodiment can be replaced with the same configuration as in the first or second embodiment.

  Further, although the ultraviolet reflectance calculation unit 405 calculates the ultraviolet reflectance using the B image and the G image, an R image may be used instead of the G image. In this case, the input to the road surface area extraction unit 404 is also a B image and an R image, not a B image and a G image.

  Further, the infrared absorption rate calculation units 112 and 311 calculate the infrared absorption rate calculation unit using the infrared image (R image) and the visible light image (G image). However, instead of the visible light image, an ultraviolet image (B Image) may be used. In that case, the input to the road surface region extraction units 108 and 310 is also an ultraviolet image instead of a visible light image.

  In the above-described embodiment, the light projection determination units 105, 306, and 407 perform determination based on the amount of solar radiation of ultraviolet rays or infrared rays. However, the light projection determination units 105, 306, and 407 may perform determination based on sunrise / sunset time data and the like. You may determine based on the amount of solar radiation estimated from the pixel value of a light image (or G image as a visible light image). For example, in the first embodiment, the light projection determination unit 105 performs determination based on the amount of solar radiation of ultraviolet rays, but this does not mean that the amount of solar radiation of infrared rays is not taken into account. Based on the premise that the amount of solar radiation is small and therefore the amount of infrared solar radiation is small if there is little, only the ultraviolet radiation amount is used for the determination for the sake of simplicity. The same applies to the case where the projection is determined based on other criteria such as the amount of solar radiation of infrared rays, and if it is possible to determine whether both ultraviolet rays and infrared rays are hitting the road surface sufficiently, whether to determine based on what is the embodiment It may be determined appropriately according to the situation. Of course, in the light projection determination part 105,306,407, you may determine according to the solar radiation amount of both an ultraviolet-ray and infrared rays. For example, a threshold value may be set for each of ultraviolet rays and infrared rays, and light may be emitted from the lighting device when both are less than the threshold value. Good. In addition to determining whether to irradiate light from the illuminating device, the amount of light radiated from the illuminating device to the road surface may be adjusted according to the amount of both solar radiation and ultraviolet radiation. In addition, the amount of solar radiation for ultraviolet rays and infrared rays may be calculated based on the pixel values in the sky region, calculated based on the pixel values in the road surface region, or may be calculated based on both pixel values. It can be appropriately determined according to the mode.

  In addition, when the road surface is irradiated with light from an illumination device such as a projector based on the determination of the light projection determination units 105, 306, and 407, the amount of solar radiation of ultraviolet rays and infrared rays is insufficient. Therefore, at this time, the ultraviolet reflectance calculation unit 104, 305 and the infrared absorption rate calculation unit 408 calculate the ultraviolet reflection rate and the infrared absorption rate based on the irradiated light amount and the pixel value of the road surface area. preferable.

  In addition, the road surface state determination apparatus of the present invention may be an apparatus housed in one housing, or may be physically separated for each part. For example, the entire road surface state determination device may be housed in a single housing and implemented as an in-vehicle device or a roadside installation device. Or the part which consists of various imaging parts and an image generation part may be installed in the road side, and the remaining part may be installed in a road management office etc. Specifically, taking the first embodiment as an example, an ultraviolet camera imaging unit 101, an ultraviolet image generation unit 102, a visible light camera imaging unit 106, a visible light image generation unit 107, an infrared camera imaging unit 109, an infrared image generation The first portion including the unit 110, the infrared thermography imaging unit 113, and the temperature distribution image generation unit 117 is installed on the road side (first place), and the road surface / sky region extraction unit 103, the ultraviolet reflectance calculation unit 104, the light projection It comprises a determination unit 105, a first road surface region extraction unit 108, a second road surface region extraction unit 111, an infrared absorption rate calculation unit 112, a road surface state determination unit 114, a road surface state image generation unit 115, and a road surface determination result output unit 116. You may install a 2nd part in a road management office etc. (2nd place). In this case, since it is necessary to transmit the image data from the first part to the second part, the first part further includes a wired or wireless data transmission means, and the second part receives the image data. Data receiving means.

  Further, in order to control an illumination device such as a projector according to the determination results of the light projection determination units 105, 306, and 407, a communication unit is provided as necessary. For example, there is a case where the light projection determination unit is in a road management office and a streetlight is used as the lighting device, or the entire road surface state determination device is implemented as an in-vehicle device and the headlamp of the vehicle is used as the lighting device. However, in any case, the determination results of the light projection determination units 105, 306, and 407 are sent to the lighting device by wired or wireless communication means to control the lighting device.

  In the above-described embodiment, the determination of the road surface state is a discrete determination as to which of the predetermined classification items (“dry”, “water film”, etc.) corresponds, but the road surface state is continuously determined. It may be expressed by a numerical value. That is, for example, in the second embodiment, the road surface state may be determined by applying a numerical value that is an output value from the neural network for road surface state determination to a discrete classification item such as “dry”. You may use the numerical value which is a continuous value as a road surface state determination result as it is. In that case, it may be determined so that the numerical value increases as the road surface becomes dangerous, and the pixel value of the road surface state image may be determined according to the value.

The modifications described above can be arbitrarily combined.
(Supplementary note 1) A road surface state determination device for determining a road surface state,
Ultraviolet feature amount calculating means for calculating an ultraviolet feature amount on the road surface based on an ultraviolet image obtained by imaging the road surface on which light including ultraviolet rays and infrared rays hits;
An infrared feature amount calculating means for calculating an infrared feature amount on the road surface based on an infrared image obtained by imaging the road surface on which the light hits;
Road surface state determining means for determining the state of the road surface based on the calculated ultraviolet feature amount, the infrared feature amount, and the temperature distribution image of the road surface;
A road surface condition judging device.
(Appendix 2) The ultraviolet image is an image including the sky and the road surface,
The ultraviolet feature amount calculating means calculates a sky pixel value corresponding to the sky region of the ultraviolet image and a road surface pixel value corresponding to the road surface region, and the ultraviolet feature based on a ratio of the sky pixel value and the road surface pixel value. Calculate the quantity,
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Supplementary Note 3) The ultraviolet feature amount calculating means uses a visible light image obtained by imaging the road surface for the calculation of the ultraviolet feature amount, and based on a ratio between a pixel value of the visible light image and a pixel value of the ultraviolet image. Calculating the ultraviolet feature amount;
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Supplementary Note 4) The ultraviolet feature amount calculating means calculates the ultraviolet feature amount based on a ratio between a pixel value of the infrared image and a pixel value of the ultraviolet image.
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Appendix 5) The infrared image is an image including the sky and the road surface,
The infrared feature amount calculating means calculates a sky pixel value corresponding to the sky region of the infrared image and a road surface pixel value corresponding to the road surface region, and the infrared feature based on a ratio of the sky pixel value and the road surface pixel value. Calculate the quantity,
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Appendix 6) The infrared feature amount calculating means uses a visible light image obtained by imaging the road surface for the calculation of the infrared feature amount, and based on a ratio between a pixel value of the visible light image and a pixel value of the infrared image. Calculating the infrared feature amount;
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Additional remark 7) The said infrared feature-value calculation means calculates the said infrared feature-value before based on ratio of the pixel value of the said ultraviolet image, and the pixel value of the said infrared image,
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Appendix 8) a first filter that blocks a blue wavelength region of visible light;
Light having a wavelength sensitivity from ultraviolet to infrared and having passed through the first filter is converted into R corresponding to the red wavelength region of visible light and G corresponding to the green wavelength region of visible light and blue of visible light. A first imaging unit that decomposes the light into three components of B corresponding to the wavelength region and ultraviolet rays, respectively receives light, and performs color imaging;
A second filter that blocks the red wavelength region of visible light;
Light having a wavelength sensitivity from ultraviolet to infrared and passing through the second filter is converted into R corresponding to the red wavelength region of visible light and G corresponding to the green wavelength region of visible light and blue of visible light. A second imaging unit that decomposes the light into three components of B corresponding to the wavelength region and ultraviolet rays, receives each of the components, and performs color imaging;
The road surface is imaged by the first imaging unit, and the ultraviolet image is generated based on a B pixel value obtained corresponding to B among the three components,
The infrared image is generated based on an R pixel value obtained by imaging the road surface with the second imaging unit and corresponding to R among the three components.
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Supplementary note 9) a filter that blocks light in the blue wavelength region and the red wavelength region of visible light;
Light having a wavelength sensitivity from ultraviolet rays to infrared rays, and the light transmitted through the filter is divided into a red wavelength region of visible light, R corresponding to infrared light, G corresponding to a green wavelength region of visible light, and a blue wavelength region of visible light, and An image pickup unit that decomposes the light into three components of B corresponding to ultraviolet rays, receives the light separately, and performs color image pickup;
The road surface is imaged by the imaging unit, and the ultraviolet image and the infrared image are generated based on the B pixel value and the R pixel value obtained respectively corresponding to B and R among the three components. ,
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Additional remark 10) It further has a temperature distribution image generation means for generating the temperature distribution image,
The temperature distribution image generation means generates the temperature distribution image based on an infrared image obtained by imaging the road surface and temperatures measured at two locations on the photographed road surface.
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Supplementary Note 11) For each divided region obtained by dividing the road surface into one or more, the road surface state determination means determines which of the predetermined road surface state classification items the road surface state of the divided region is classified into. The road surface condition determination device according to appendix 1, wherein the determination is performed.
(Additional remark 12) The said road surface state determination means determines the road surface state of this area | region for every division area which divided | segmented the said road surface,
The road surface state determination device further includes road surface state image generation means for generating a road surface state image representing the state of the road surface,
The road surface state image generating means determines a height and / or color on the image as a road surface expression representing each divided region on the road surface state image according to the road surface state of each divided region, and the divided region Generating the road surface state image based on the road surface representation and the visible light image of the road surface,
The road surface condition determination apparatus according to Supplementary Note 1, wherein
(Supplementary note 13) A road surface condition determination method,
Calculate the ultraviolet feature amount on the road surface based on the ultraviolet image obtained by imaging the road surface where light including ultraviolet rays and infrared rays hits,
Calculating an infrared feature amount on the road surface based on an infrared image obtained by imaging the road surface on which the light hits;
A road surface state determination method, wherein the road surface state is determined based on the calculated ultraviolet feature amount, the infrared feature amount, and a temperature distribution image of the road surface.

It is a figure which shows the principle of this invention. It is a functional block diagram of a road surface condition judging device by a first embodiment. It is a figure which shows the example of the determination conditions of a road surface state. It is a figure which shows the example of a road surface state determination result image. It is a functional block diagram of the road surface condition determination apparatus by 2nd embodiment. It is a figure explaining the principle which produces | generates an infrared image and an ultraviolet-ray image using a stereo color camera. It is a figure which shows the example of the pixel arrangement configuration of an image. It is a figure which shows the structural example of a neural network. It is a functional block diagram of the road surface condition determination apparatus by 3rd embodiment. It is a figure explaining the principle which produces | generates an infrared image and an ultraviolet-ray image using a triple band pass filter.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Road surface state determination apparatus 2 Ultraviolet feature amount calculation means 3 Infrared feature amount calculation means 4 Road surface state determination means 101 Ultraviolet camera imaging part 102 Ultraviolet image generation part 103 Road surface and sky area extraction part 104 Ultraviolet reflectance calculation part 105 Light projection determination part DESCRIPTION OF SYMBOLS 106 Visible light camera imaging part 107 Visible light image generation part 108 1st road surface area extraction part 109 Infrared camera imaging part 110 Infrared image generation part 111 2nd road surface area extraction part 112 Infrared absorptance calculation part 113 Infrared thermography imaging part 114 Road surface state determination unit 115 Road surface state image generation unit 116 Road surface determination result output unit 117 Temperature distribution image generation unit 301 Blue wavelength region cut unit 302 First imaging unit 303 First image generation unit 304 Road surface / sky region extraction unit 305 Ultraviolet light Reflectance calculation unit 306 Light projection determination unit 307 Red wave Long region cut unit 308 Second imaging unit 309 Second image generation unit 310 Road surface region extraction unit 311 Infrared absorption rate calculation unit 312 Relative position calculation unit 313 Infrared thermography imaging unit 314 Road surface state determination unit 315 Road surface state image generation unit 316 Color image composition unit 317 Road surface determination result output unit 318 Temperature distribution image generation unit 401 Blue wavelength region / red wavelength region cut unit 402 Imaging unit 403 Image generation unit 404 Road surface region extraction unit 405 Ultraviolet reflectance calculation unit 406 Road surface / sky region extraction Unit 407 light projection determination unit 408 infrared absorption rate calculation unit 409 infrared image imaging unit 410 temperature measurement unit 411 temperature distribution image generation unit 412 road surface state determination unit 413 road surface state image generation unit 414 road surface determination result output unit 501 input layer 502 intermediate layer 503 Output layer

Claims (5)

A road surface state determination device for determining a road surface state,
Ultraviolet feature amount calculating means for calculating an ultraviolet feature amount on the road surface based on an ultraviolet image obtained by imaging the road surface on which light including ultraviolet rays and infrared rays hits;
An infrared feature amount calculating means for calculating an infrared feature amount on the road surface based on an infrared image obtained by imaging the road surface on which the light hits;
Road surface state determining means for determining the state of the road surface based on the calculated ultraviolet feature amount, the infrared feature amount, and the temperature distribution image of the road surface;
A road surface condition judging device. The ultraviolet image is an image including the sky and the road surface,
The ultraviolet feature amount calculating means calculates a sky pixel value corresponding to the sky region of the ultraviolet image and a road surface pixel value corresponding to the road surface region, and the ultraviolet feature based on a ratio of the sky pixel value and the road surface pixel value. Calculate the quantity,
The road surface state determination apparatus according to claim 1. The infrared feature amount calculation means uses a visible light image obtained by imaging the road surface for the calculation of the infrared feature amount, and the infrared feature amount based on a ratio of a pixel value of the visible light image and a pixel value of the infrared image. To calculate,
The road surface state determination apparatus according to claim 1. A temperature distribution image generating means for generating the temperature distribution image;
The temperature distribution image generation means generates the temperature distribution image based on an infrared image obtained by imaging the road surface and temperatures measured at two locations on the photographed road surface.
The road surface state determination apparatus according to claim 1. A road surface condition determination method,
Calculate the ultraviolet feature amount on the road surface based on the ultraviolet image obtained by imaging the road surface where light including ultraviolet rays and infrared rays hits,
Calculating an infrared feature amount on the road surface based on an infrared image obtained by imaging the road surface on which the light hits;
A road surface state determination method, wherein the road surface state is determined based on the calculated ultraviolet feature amount, the infrared feature amount, and a temperature distribution image of the road surface.