JP6901647B1 - Visibility estimation device, visibility estimation method, and recording medium - Google Patents

Abstract

Provided is a visibility estimation device and the like capable of estimating visibility without the need for a special measuring device. An image of the outdoors taken by the photographing means C is acquired (S1), an ambient light and light transmission map is estimated from the image, and a sharpened image obtained by sharpening the image is generated based on the ambient light and light transmission map. (S2), the object is recognized from the sharpened image, the pixel scale of the object is converted into the physical scale of the object based on the information about the actual scale of the recognized object (S5), and the physical of the object. The depth direction distance of the object is calculated from the depth direction component of the scale (S6), and the relational expression between the transmission map and the distance from the photographing means is applied to the image to calculate the light attenuation parameter of the atmosphere (S8, S20). ), The field of view is calculated from the depth direction distance based on the relational expression to which the calculated light attenuation parameters are applied (S8, S21).

Description

The present invention relates to a visibility estimation device by image processing, a visibility estimation method, and a recording medium.

Visibility is measured as a measure of poor visibility due to bad weather conditions such as fog, mist, rain, and snow, cloudy weather, and illuminance conditions at night. For example, in Patent Document 1, the light source irradiates near infrared rays toward the measurement target space, the spectroscopic camera is equipped with a polarizing filter, and among the near infrared rays polarized by the object to be measured in the measurement target space, the polarizing filter By transmitting near infrared rays that are the same as the polarization angle, a two-dimensional image of the object to be measured is imaged, and the image analyzer correlates with different spectral / polarized image information of the object to be measured in the measurement target space based on the two-dimensional image information. A meteorological measuring device that accurately identifies the type of the object to be measured in the measurement target space by equal statistical processing and calculates the amount and visibility is disclosed.

Japanese Unexamined Patent Publication No. 2012-86604

However, in the above technology, near infrared rays are irradiated from a light source and the visibility is measured by the reflected light, so that a special measuring device is required, and the cost increases especially when it is installed at many observation points. Was there.

Therefore, the present invention has been made in view of the above problems and the like, and an example of the problem is to provide a visibility estimation device and the like capable of estimating visibility without the need for a special measuring device.

In order to solve the above problems, the invention according to claim 1 comprises an acquisition means for acquiring an image of an outdoor photographed by a photographing means, an estimation means for estimating ambient light and a transmission map of light from the image, and an estimation means. Based on the ambient light and the transmission map, the image sharpening means for generating a sharpening image obtained by sharpening the image, and the information regarding the actual scale of the recognized object by recognizing the object from the sharpening image. Based on this, an object scale conversion means that converts the pixel scale of the object into the physical scale of the object, and a depth direction distance calculation means that calculates the depth direction distance of the object from the depth direction component of the physical scale of the object. The light attenuation parameter calculating means for calculating the light attenuation parameter indicating the degree of light attenuation by applying the relational expression between the transmission map and the distance from the photographing means to the image, and the calculated light attenuation parameter. Based on the relational expression to which the above is applied, it is characterized in that it is provided with a visibility calculation means for calculating the visibility from the distance in the depth direction.

Further, according to the second aspect of the present invention, when the visibility estimation device according to the first aspect cannot recognize the object for calculating the depth direction distance in the image, the height of the installation of the photographing means and the height of the installation of the photographing means. It is characterized in that the distance in the depth direction in the image is calculated from the shooting direction.

Further, according to the invention of claim 3, in the vision estimation device according to claim 1 or 2, the relational expression is an element of at least one of a sine function and a cosine function in addition to the exponential function. The light attenuation parameter includes a parameter for an element of at least one of the sine function and the cosine function calculated by the light attenuation parameter calculation means.

Further, in the invention according to claim 4, in the visibility estimation device according to any one of claims 1 to 3, the visibility calculation means applies the calculated light attenuation parameter to the relational expression. It is a value of light transmission based on the above, and is characterized in that the visibility is calculated from the value of light transmission in a predetermined region of the image and the distance in the depth direction.

Further, in the invention according to claim 5, in the visibility estimation device according to any one of claims 1 to 4, the visibility calculation means applies the calculated light attenuation parameter to the relational expression. It is characterized in that the visibility at which the value of light transmission based on the above becomes a predetermined value is calculated.

The invention according to claim 6 is characterized in that the visibility estimation device according to any one of claims 1 to 5 further includes a visibility level means for converting the visibility into a visibility level. To do.

The invention according to claim 7 includes an acquisition step in which the acquisition means acquires an image of the outdoors photographed by the photographing means, and an estimation step in which the estimation means estimates ambient light and a transmission map of light from the image. An image sharpening step in which the image sharpening means generates a sharpening image obtained by sharpening the image based on the ambient light and the transmission map, and an object scale conversion means recognizes an object from the sharpening image. Then, based on the information about the actual scale of the recognized object, the object scale conversion step of converting the pixel scale of the object to the physical scale of the object and the depth direction distance calculation means are the physical scales of the object. The depth direction distance calculation step of calculating the depth direction distance of the object from the component in the depth direction and the light attenuation parameter calculation means apply the relational expression between the transmission map and the distance from the photographing means to the image. Based on the light attenuation parameter calculation step for calculating the light attenuation parameter indicating the degree of light attenuation and the relational expression to which the calculated light attenuation parameter is applied, the visibility calculation means calculates the visibility from the depth direction distance. It is characterized by including a visibility calculation step to be performed.

The invention according to claim 8 is an acquisition means for acquiring an image of an outdoor photographed by a computer, an estimation means for estimating an ambient light and a transmission map of light from the image, the ambient light and the transmission. An image sharpening means for generating a sharpened image obtained by sharpening the image based on a map, recognizing an object from the sharpened image, and based on information on the actual scale of the recognized object, the pixel scale of the object. Object scale conversion means for converting the physical scale of the object from, object depth direction distance calculation means for calculating the depth direction distance of the object from the depth direction component of the physical scale of the object, the transmission map and the photographing means. To the light attenuation parameter calculation means for calculating the light attenuation parameter indicating the degree of light attenuation by applying the relational expression with the distance from the image to the image, and the relational expression to which the calculated light attenuation parameter is applied. Based on this, it is characterized in that it functions as a visibility calculation means for calculating a visibility from the distance in the depth direction.

According to the present invention, an image taken outdoors by a photographing means is acquired, a sharpened image is generated by sharpening the image, an object is recognized from the sharpened image, and information on the actual scale of the recognized object is obtained. Based on, the pixel scale of the object is converted into the physical scale of the object, the depth direction distance is calculated from the depth direction component of the physical scale of the object, and the relationship between the light transmission map and the distance from the photographing means. By applying the formula to the image, calculating the light attenuation parameter of the atmosphere, and calculating the visibility from the depth direction distance based on the relational expression to which the calculated light attenuation parameter is applied, no special measuring device is required. , The visibility can be estimated.

It is a schematic diagram which shows the outline structure example of the visibility estimation system which concerns on embodiment of this invention. It is a block diagram which shows the outline configuration example of the information processing server apparatus of FIG. It is a schematic diagram which shows an example of the data stored in the image database of the information processing server apparatus of FIG. It is a schematic diagram which shows an example of the data stored in the object database of the information processing server apparatus of FIG. It is a schematic diagram which shows an example of the data stored in the camera information database of the information processing server apparatus of FIG. It is a block diagram which shows the outline configuration example of the terminal apparatus of FIG. It is a flowchart which shows an example of the image processing which calculates the visibility. It is a schematic diagram which shows an example of the image taken by a camera. It is a schematic diagram which shows an example of the partial image area of an object. It is a schematic diagram which shows an example of the partial image area of an object. It is a schematic diagram which shows an example of the partial image area of an object. It is a schematic diagram which shows an example of the image taken by a camera. It is a schematic diagram which shows an example of the distance in the depth direction. It is a schematic diagram which shows an example of the distance in the depth direction. It is a schematic diagram which shows an example of the distance in the depth direction. It is a schematic diagram which shows an example of the image taken by a camera. It is a schematic diagram which shows an example of the image taken by a camera. It is a schematic diagram which shows an example of a camera system model. It is a schematic diagram which shows an example of an image and a distance in a depth direction. It is a schematic diagram which shows an example of an image and a distance in a depth direction. It is a schematic diagram which shows an example of the visibility level. It is a flowchart which shows the subroutine of image sharpening of FIG. It is a flowchart which shows the subroutine of the visibility estimation of FIG. It is a schematic diagram which shows an example of the calculation of visibility. It is a schematic diagram which shows an example of the calculation of visibility. It is a schematic diagram which shows an example of a camera system model.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are embodiments when the present invention is applied to the visibility estimation system.

[1. Overview of visibility estimation system configuration and functions]

First, the configuration and outline function of the visibility estimation system according to the embodiment of the present invention will be described with reference to FIG.
FIG. 1 is a schematic diagram showing a schematic configuration example of a visibility estimation system according to an embodiment of the present invention.

As shown in FIG. 1, the visibility estimation system 1 includes an information processing server device 10 that calculates weather information such as visibility at each point from image data from a camera C installed at each point such as a road, and information processing. It includes a terminal device 20 that displays weather information such as visibility provided by the server device 10.

The information processing server device 10 which is an example of the visibility estimation device and each terminal device 20 are connected to each other through a network 3. The camera C, which is an example of the photographing means, photographs the outdoor weather conditions such as roads. The camera C communicates with the information processing server device 10 by wireless communication through the radio station 5. Further, the network 3 may be, for example, the Internet, a dedicated communication line (for example, a CATV (Community Antenna Television) line), or a mobile communication network (including a base station or the like).

The information processing server device 10 is a server device of a company that provides various weather information. The information processing server device 10 provides the terminal device 20 with weather information of transportation such as highways, general roads, railways, and airports, and weather information of routes, ports, and seas. For example, as an example of meteorological information, information such as weather, temperature, humidity, atmospheric pressure, wind direction, wind speed, and visibility indicating the degree of visibility can be mentioned.

Further, the information processing server device 10 may provide traffic information such as vehicle congestion and traffic regulation, operation status of trains, airplanes, etc., and operation status of ships.

Further, the information processing server device 10 analyzes the image data from the camera C at each point and calculates a value related to the weather such as visibility. Further, the information processing server device 10 may perform image analysis of image data from the camera C to calculate and provide a road surface condition, a runway condition, a sea condition, a traffic volume, and the like.

The terminal device 20 receives the weather information from the information processing server device 10 and displays the weather information.

The camera C is a color or black-and-white digital camera having a photographing element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera C captures a still image, a moving image, or the like. The camera C may be a camera mounted on a moving body such as a vehicle, an airplane, or a train. The camera C may be a camera mounted on a mobile terminal such as a smartphone.

Each camera C is installed at a predetermined interval or at a predetermined position on a road such as an expressway. A pole is installed on the side of the road, and a camera C is installed at a predetermined height. The camera C is oriented in the traveling direction of the road, for example. It suffices if the camera C is installed so that the photograph can be taken from above the road.

[2. Configuration and functions of each information processing server device]
(2.1 Configuration and function of information processing server device 10)
Next, the configuration and function of the information processing server device 10 will be described with reference to FIGS. 2 to 5.

FIG. 2 is a block diagram showing a schematic configuration example of the information processing server device 10. FIG. 3 is a schematic diagram showing an example of data stored in the image database of the information processing server device 10. FIG. 4 is a schematic diagram showing an example of data stored in the object database of the information processing server device 10. FIG. 5 is a schematic diagram showing an example of data stored in the camera information database of the information processing server device.

As shown in FIG. 2, the information processing server device 10 that functions as a computer includes a communication unit 11, a storage unit 12, a display unit 13, an operation unit 14, an input / output interface unit 15, and a control unit 16. It has. The control unit 16 and the input / output interface unit 15 are connected to each other via the system bus 17.

The communication unit 11 is connected to the network 3 to control the communication state with each camera C and each terminal device 20.

The storage unit 12 is composed of, for example, a hard disk drive, a silicon disk drive, or the like, and stores various programs such as an operating system and a server program, weather information, and the like. The various programs may be acquired from another server device or the like via the network 3, or may be recorded on a recording medium and read via the drive device.

Further, the storage unit 12 includes an image database 12a (hereinafter referred to as "image DB 12a"), an object information database 12b (hereinafter referred to as "object information DB 12b"), and a machine learning database 12c (hereinafter referred to as "machine learning DB 12c"). ), Meteorological / traffic information database 12d (hereinafter referred to as "meteorological / traffic DB 12d"), camera information database 12e (hereinafter referred to as "camera information DB 12e"), and the like are constructed.

Image data taken by each camera C is stored in the image DB 12a. As shown in FIG. 3, the image DB 12a stores the time taken in association with the camera ID that identifies the camera C and the image data in a predetermined format.

The position information of the camera C may be stored in another database in association with the camera ID in the storage unit 12.

The object information DB 12b stores the size information of the object for each type of the object as an example of the information regarding the actual scale of the recognized object. For example, as shown in FIG. 4, the object information DB 12b stores the object type and the size information of the object in association with the object type ID that specifies the type of the object. Examples of the size information of the object include a total length [m] × height [m] × width “m”.

Here, as an example of the type of the object, in the case of a vehicle, the type such as an ordinary automobile, a light automatic vehicle, a bus, a small bus, a large truck, and a medium-sized truck can be mentioned. As an example of the type of object, in the case of an aircraft, there are types such as passenger aircraft and Cessna aircraft. As an example of the type of an object, in the case of a landmark such as a building, the type of a small building, a medium-sized building, or the like can be mentioned. An object type ID may be set for each vehicle type of each automobile manufacturer.

The machine learning DB 12c stores data necessary for analysis by AI (Artificial Intelligence). In the case of machine learning for object recognition, learned data for object recognition is stored. For example, it is the trained data for object recognition learned by the image for each type of the object which is the teacher data. An object type ID is assigned to the object type.

The meteorological / traffic DB 12d stores meteorological data such as temperature, humidity, atmospheric pressure, wind direction, wind speed, weather, water temperature, waves, sea current, tidal current, atmospheric pressure arrangement, and various image data from satellites. These meteorological data are obtained from each ship, meteorological satellite, meteorological organization of each country, observation point, and the like. Further, in the meteorological / traffic DB 12d, in addition to the past meteorological data, the calculated predicted meteorological data is also stored. For example, the expected path of a low pressure system, changes in atmospheric pressure, and the like can be mentioned.

The weather / traffic DB 12d stores traffic information such as vehicle congestion and traffic restrictions, operation status of trains and airplanes, and operation status of ships. Further, the weather / traffic DB 12d stores map information including roads, railroads, nautical charts, and the like. In the case of ships, the map information may include information such as the location of icebergs, avoidance areas such as areas where pirates infest, and emission control areas.

The camera information DB 12e stores information such as the position where each camera C is installed. As shown in FIG. 5, in the image DB 12a, the position information (for example, longitude and latitude) of the place where the camera C is installed and the height of the camera C at the installation point are associated with the camera ID that identifies the camera C. Now, the shooting direction of the camera C and the like are stored. The height of the camera C may be the height from the shooting center of the camera C to the installation surface, or the height of the pole that supports the camera C. The shooting direction of the camera C is represented by, for example, an azimuth angle, an elevation angle, or the like. When the direction of the camera is variable, the data of the shooting direction of the camera at the time of shooting may be stored in the image DB 12a.

The display unit 13 is composed of, for example, a liquid crystal display element, an organic EL (Electro Luminescence) element, or the like. The operation unit 14 is composed of, for example, a keyboard, a mouse, and the like. The input / output interface unit 15 performs interface processing between the communication unit 11 and the like and the control unit 16.

The control unit 16 is composed of a CPU (Central Processing Unit) 16a, a ROM (Read Only Memory) 16b, a RAM (Random Access Memory) 16c, and the like.

(2.2 Configuration and function of terminal device 20)
Next, the configuration and function of the terminal device 20 will be described with reference to FIG.
FIG. 6 is a block diagram showing an example of an outline configuration of the terminal device 20.

As shown in FIG. 6, the terminal device 20 that functions as a computer includes a communication unit 21, a storage unit 22, a display unit 23, an operation unit 24, an input / output interface unit 25, and a control unit 26. ing. The control unit 26 and the input / output interface unit 25 are connected via the system bus 27. Since the terminal device 20 has almost the same configuration and function as the information processing server device 10, the description of common parts will be omitted.

The communication unit 21 is connected to the network 3 to control the communication state with the information processing server device 10.

The storage unit 22 is composed of, for example, a hard disk drive, a silicon disk drive, or the like, like the storage unit 12 of the information processing server device 10.

The display unit 23 corresponds to the display unit 13, the operation unit 24 corresponds to the operation unit 14, and the input / output interface unit 25 corresponds to the input / output interface unit 15. Further, the control unit 26 is composed of a CPU 26a, a ROM 26b, a RAM 26c, and the like corresponding to the control unit 16.

[3. Operation of visibility estimation system 1]
Next, the operation of the visibility estimation system 1 according to the first embodiment of the present invention will be described with reference to the drawings.

(3.1 Operation of visibility estimation system 1)
The operation of the visibility estimation system 1 will be described with reference to FIGS. 7 to 14.
FIG. 7 is a flowchart showing an example of image processing for calculating visibility. FIG. 8 is a schematic view showing an example of an image taken by the camera. 9A to 9C are schematic views showing an example of a partial image region of an object. 10B to 10D are schematic views showing an example of a distance in the depth direction. 11A and 11B are schematic views showing an example of an image taken by the camera. FIG. 12 is a schematic diagram showing an example of a camera system model. 13A and 13B are schematic views showing an example of an image and a distance in the depth direction. FIG. 14 is a schematic view showing an example of the visibility level.

As shown in FIG. 7, the visibility estimation system 1 acquires and stores image data (step S1). Specifically, the information processing server device 10 acquires an image taken by each camera C installed outdoors from each camera C. More specifically, the control unit 16 receives image data from each camera C together with the camera ID of each camera C. The image data may include data at the shooting time.

The control unit 16 stores the received image data in the image DB 12a based on the camera ID.

The information processing server device 10 functions as an example of the acquisition means for acquiring an image captured outdoors by the photographing means.

Next, the visibility estimation system 1 performs image sharpening processing (step S2). Specifically, the information processing server device 10 acquires an image from the image DB 12a and performs an image sharpening process when the image is not clear due to fog, haze, rain, or the like. More specifically, the control unit 16 estimates the ambient light and the transmission map of the light from the acquired image, and generates a sharpened image in which the image is sharpened based on the ambient light and the transmission map. Fog and haze are the same meteorological phenomenon, and the visibility of fog is closer than that of haze. Details will be described in the image sharpening processing subroutine described later.

Here, in order to remove fog and haze, for example, an image sharpening process is performed using a physical model. The physical model is, for example, the model of equation (1).
I (x) = J (x) t (x) + A (1-t (x)) ... (1)
Here, the original image I (x) with fog and the like taken by the camera C, the sharpened image J (x) without fog and the like, the intensity A of the ambient light, and the light transmission map t (x). .. The light transmission map is also called an optical transmission map, an optical attenuation function, or an optical transfer function. Further, the vector x is a two-dimensional position vector (x, y) in the image, and is, for example, I (x, y), J (x, y), t (x, y).

In this physical model (also called a linear image generation model or a fog image model), two optical paths are assumed for the brightness (I (x)) of the image, and one is on the image from the sun or the moon. There are two types of objects: one with light attenuation due to fog or mist (J (x) t (x)) and one with light directly entering the image (A (1-t (x))). In the present embodiment, as shown in FIG. 8, the lower left of the image is set as the origin. The vector x = 0 is the lower part of the image and corresponds to the part closest to the camera C in a normal scene.

It is assumed that the transparency map t (x) changes exponentially according to the position in the image.
t (x) = exp (-beta x d (x)) ... (2)
Here, let d (x) be the distance from the camera viewpoint to the depth direction in the image. beta is a parameter that adjusts the degree of light attenuation and is an example of light attenuation parameter. The light attenuation parameter is a coefficient indicating the degree of light attenuation due to an aerosol or the like. Alternatively, the light attenuation parameter may be a light scattering coefficient. Equation (2) is an example of the relational expression between the transparent map and the distance from the photographing means.

The transmission map t (x) corresponds to the density of fog in the image. The farther away from the camera C, the higher the density, and the farther in front, the lower the fog density. Further, the vector x is a two-dimensional position vector (x, y) in the image, for example, t (x, y), d (x, y).

If the captured image is free of fog and haze and the image is clear, the information processing server device 10 may perform the next object recognition process without performing the image sharpening process.

The information processing server device 10 functions as an example of an estimation means for estimating ambient light and a light transmission map from the image. The information processing server device 10 functions as an example of an image sharpening means for generating a sharpening image obtained by sharpening the image based on the ambient light and the transmission map.

Next, the visibility estimation system 1 recognizes the object (step S3). Specifically, the control unit 16 of the information processing server device 10 recognizes an object such as an automobile or a freight vehicle by machine learning image recognition processing with reference to the machine learning DB 12c, and determines the type of the object. .. For example, as shown in FIG. 8, the control unit 16 recognizes an object on the road in the image 30 and identifies the object type ID of each object. In FIG. 8, the recognized object is shown by a rectangular frame in the image 30. The size of the image 30 is p × q pixels. For example, as a machine learning method, CNN (Convolution Neural Network), Faster RCNN (Region Convolution Neural Network), SSD (Single Shot MultiBox Detector) and the like can be mentioned.

When the object is recognized (step S3; YES), the visibility estimation system 1 calculates the size of the object portion in the image (step S4). For example, as shown in FIGS. 9A to 9C, the control unit 16 has an object such as a partial image area 30a of an object of an ordinary automobile, a partial image area 30b of an object of a large truck, and a partial image area 30c of an object of an ordinary automobile. Extract the partial image area of. The control unit 16 measures the height ha × width wa × total length da of the object in pixel units in the partial image region 30a. The control unit 16 measures the height hb × width wb × total length db of the object in pixel units in the partial image region 30b. The control unit 16 measures the height hc × width wc × total length dc of the object in pixel units in the partial image region 30c. For example, the control unit 16 extracts an edge from the partial image area 30a, fits a rectangular parallelepiped model, and measures the height × width × total length of the object in pixel units. The control unit 16 also calculates the orientation of each object.

By machine learning image processing, the control unit 16 may calculate the height × width × total length of the object in pixel units, or may calculate the orientation of the object. As the size of the object portion in the image, the control unit 16 may calculate the pixels in the x-direction and the y-direction of each partial image region. For example, the size of the partial image area 30a is xa × ya pixels, the size of the partial image area 30b is xb × yb pixels, and the size of the partial image area 30c is xc × yc pixels. Further, after the control unit 16 performs three-dimensional rotation image processing on the partial image areas 30a, 30b, and 30c so that the surface of the object corresponding to the height × width faces the front, the height of the object hc × width wc × total length dc may be obtained.

When the direction of the object is substantially the same as the shooting direction of the camera C, the total length Da, Db, Dc, etc. are examples of the components in the depth direction of the physical scale of the object. The control unit 16 may calculate the component in the depth direction of the physical scale of the object from the total length Da, the height Ha, the width Wa, etc. from the angle between the direction of the object and the shooting direction of the camera C.

Next, the visibility estimation system 1 calculates the physical scale of the object (step S5). Specifically, the control unit 16 refers to the object information DB 12b based on the specified object type ID, and extracts the overall length [m] × height [m] × width “m” as the object size information. To do.

The information processing server device 10 recognizes an object from the sharpened image and converts the pixel scale of the object into the physical scale of the object based on the information about the actual scale of the recognized object. It functions as an example.

The control unit 16 calculates the ratio between the pixel scale and the physical scale of the object. As shown in FIG. 9A, the control unit 16 has a pixel scale height ha × width wa × total length da and a physical scale height Ha × width Wa × total length Da, that is, ratio Ha / ha, ratio Wa /. wa, ratio Da / da is calculated. This ratio represents the physical scale [m] per pixel.

The average value of the ratio Ha / ha, the ratio Wa / wa, and the ratio Da / da may be used as the ratio between the pixel scale and the physical scale of the object. Further, when the control unit 16 averages, the corrected value may be averaged based on the orientation of the object in the image 30. As the correction, the correction may be performed in which each side of the height × width × total length of the object in the partial image area is rotated so as to face the front of the camera C. Further, when it is difficult to measure the total length of the object, only the ratio of height and width may be used. Instead of the height ha x width wa of the pixel scale, the size xa x ya of the partial image area may be used.

When there are a plurality of objects, the control unit 16 may calculate the ratio of the pixel scale of the plurality of objects to the physical scale of the objects. The control unit 16 may mainly calculate the ratio of the pixel scale to the physical scale of the object with respect to the object closer to the camera C.

When the front or back surface of the object faces the camera C, the ratio Da / da, the ratio Db / db, and the ratio Dc / dc are the physical scales per pixel with respect to the pixel position in the predetermined y direction of the image 30. The distance in the depth direction. The physical scale depth direction distance per pixel is the ratio Ha / of wa'obtained by rotationally converting the object in the image so that the width direction of the object is the depth instead of the ratio Da / da. It may be wa'. Here, the distance in the depth direction may be the distance in the y direction, the distance along the surface of the road, or the distance from the viewpoint of the camera C in the image 30.

Next, the visibility estimation system 1 estimates the distance in the depth direction (step S6). For example, as shown in FIG. 10A, the control unit 16 is based on the center coordinates of each object such as the center coordinates (xa0, ya0) of the partial image area 30a and the center coordinates (xc0, yc0) of the partial image area 30c. The depth direction distance (for example, y1 = yc0-ya0) of each object on the pixel scale is calculated. As shown in FIG. 10B, the control unit 16 determines the distance in the depth direction of the physical scale per pixel from y = ya0 to y = yc0 from the ratio Da / da at ya0 and the ratio Dc / dc at yc0. Is added to estimate the depth direction distance dy1 between the objects as shown in FIG. 10B.

As shown in FIG. 10B, the control unit 16 may calculate the distance in the depth direction of the object from the ratio Wa / wa, the ratio Wc / wc, and y1. Here, it is assumed that this ratio changes from y = ya0 to y = yc0 from the ratio Wa / wa to the ratio Wc / wc proportionally or by a predetermined increasing function. As shown in FIG. 10C, when the object is long, the distance in the depth direction of the object may be length Db or length dy3.

Further, in FIG. 10B, in the object in the partial image area 30a and the object in the partial image area 30c, the size of the object is normalized, and a number of objects are packed in consideration of the similarity multiple law between the objects. The control unit 16 may estimate the distance in the depth direction based on the number of objects × the total length [m].

Further, as shown in FIG. 10D, the control unit 16 may estimate the distance in the depth direction from the apparent length according to the similarity double rule. For example, assuming that the width of the vehicle is almost constant (even if the depth estimation accuracy allows an error of several tens of meters, when the ratio Wa / wa and the ratio Wc / wc are normalized), the depth direction is the depth direction of the road. The width length Wa and Wc of the front surface of the vehicle are selected, and the control unit 16 calculates Wa (for example, 2 m) and Wc × wc / wa (for example, 1 m) as the apparent width length. To do.

Next, the total length of the vehicle is set to about 5 m, and the control unit 16 estimates the depth by the similarity multiple rule. For example, the visibility can be determined from the apparent total length Da (for example, 5 m) and the apparent total length Da'= Da × wa / wc (for example, 10 m) between two vehicles having an apparent width length of 2 m and an apparent width length of 1 m. It is estimated that 5m + 10m = 15m. If the objects do not have the same object type ID, the size is normalized. For example, the control unit 16 may calculate using the reciprocal of the ratio Wa / wa and the ratio Wc / wc instead of wa and wc. In this case, for example, the apparent overall length Dc'= Dc × wa / wc × Wc / Wa.

It should be noted that the depth direction distance between the objects and the depth direction distance from the camera C to each object may be calculated by modeling in 3D by image processing using AI.

The information processing server device 10 also functions as an example of a depth direction distance calculating means for calculating the depth direction distance of the object from the depth direction component of the physical scale of the object.

When the object cannot be recognized (step S3; NO), the visibility estimation system 1 calculates the depth direction distance by the camera system model (step S7). The case where the object cannot be recognized is, for example, a case where an object such as a vehicle is not shown in the images 31 and 32 as shown in FIGS. 11A and 11B. The information processing server device 10 calculates the distance in the depth direction of the physical scale according to the following equation (3).

The depth direction distance dv in the camera system model is given by the equation (3).
dv = 1 / (v−vh) × H × f / cos (theta) ・ ・ ・ (3)

Here, as shown in FIG. 12, in the camera system model, the camera center C0 of the camera C, the length H of the pole supporting the camera C (from the installation point S of the camera C to the camera center C0), and the inclination theta of the pole are used. is there. It is assumed that the image plane P corresponding to the images 31 and 32 is at the focal length f from the camera center C0. As shown in FIG. 12, the depth direction distance dv is the horizontal distance from the installation point S, and the vector v is the image plane P with y = q as the base point as shown in FIGS. 11A, 11B and 12. , A vector in the direction of y = 0. The vector vh is a vector in the direction of y = 0 with y = q as the base point up to the image point corresponding to infinity in the image. d0 is the depth distance at the point of the image plane P corresponding to y = 0 of the images 31 and 32. Here, the length H is an example of the height at which the photographing means is installed. The tilt theta of the pole is an example of the shooting direction of the shooting means. The depth direction distance dv is an example of the depth direction distance in the image.

As shown in FIGS. 13A and 13B, the relationship between the vector v and the depth direction distance dv is such that in the images 31 and 32, as shown in the equation (3), the closer the vector v is to vh, the more the depth direction distance dv becomes. As v becomes longer and y = 0 approaches, the distance dv in the depth direction becomes shorter.

Next, the visibility estimation system 1 estimates the visibility (step S8). The information processing server device 10 applies, for example, the relational expression (2) between the transmission map t (x) and the distance to the image 30 (I (x)), and the light attenuation parameter beta indicating the degree of light attenuation. Is calculated, and the visibility is estimated from the distance in the depth direction based on the relational expression (2) by the calculated light attenuation parameter beta. Here, the distance in the depth direction is the distance estimated in step S6 or the distance calculated in step S7. The details will be described in the visibility estimation subroutine described later.

As shown in FIG. 14, the information processing server device 10 may express the estimated visibility on the physical scale at the visibility level. For example, when the visibility level is level 1, the visibility is 10 m or less and the front cannot be seen well. Therefore, in the case of a road, it is a level for traffic regulation. When the visibility level is level 2, the visibility is 10 m to 50 m, the front is considerably difficult to see, and in the case of a vehicle, speed regulation is performed. When the visibility level is level 3, the visibility is 50 m to 100 m, which is a level that calls attention. By dividing the visibility into levels in this way, it is possible to absorb the error of the visibility estimation and express the visibility. The level may be set arbitrarily by the user. In the case of an airfield, the visibility level is, for example, levels 1 to 5, and the visibility may be in units of 100 m.

Next, the visibility estimation system 1 outputs the visibility (step S9). Specifically, the information processing server device 10 transmits information on the visibility or visibility level estimated from the image of each camera C to the terminal device 20 together with the position information of each camera C.

The terminal device 20 displays the visibility information on the display unit 23 based on the position information on the map. The map information may be acquired from the storage unit 22 of the terminal device 20, or may be acquired from the outside of the information processing server device 10 or the like.

The information processing server device 10 calculates the unclearness of fog or the like from the image 30, and in the case of fine weather, it is not necessary to calculate the visibility described above. For example, when the shading value (for example, brightness) of an image is 100 (0 to 255 steps) or more and the deviation value is a predetermined value or more, the control unit 16 determines that the weather is fine. In addition, the predetermined value of the deviation value may be statistically determined from the image which classified whether the weather is fine or not by using the past data.

(3.2 Image sharpening subroutine)
Next, the image sharpening subroutine will be described with reference to FIG.
FIG. 15 is a flowchart showing a subroutine for sharpening an image.

Based on the equation (1), the information processing server device 10 calculates the sharpened image J (x) from the image I (x) taken by the camera C as follows.

As shown in FIG. 15, the information processing server device 10 determines the intensity of ambient light (step S10). Here, in the case of a color image (for example, an RGB image) whose brightness is represented by 8 bits (0 to 255) and a natural image, the image has various brightness and darkness. On the other hand, in a predetermined image region (patch), there are pixels having a low brightness value due to the presence of shadows, black objects, and the like. The minimum value of brightness is 0. If sunlight is present and the sky is present in the image, it is assumed that 0.1% of the brightest pixels in each RGB channel or patch correspond to the sky. The control unit 16 estimates the intensity A of the ambient light from the pixels of the corresponding portion in the image I (x).

Next, the information processing server device 10 estimates the light transmission map t (x) (step S11). For example, in the equation (1), when the brightness of the dark channel is 0 in the true image J (x) as the first approximation,
I (x) = A (1-t (x)) ... (4)
To get. Here, I (x) indicates the component I ^dark (x) of the dark channel. From this equation (4), the transparency map is
t (x) = 1-I (x) / A ... (5)
Can be transformed. However, the value <1 of 0 <t (x).

In many fields, the sky area of the fog image is considered to be infinity when viewed from the camera C. Therefore, in this sky area, the value of t (x) becomes almost 0, and I≈A, that is, the sky image. I is equal to the intensity A of the ambient light.

Next, the information processing server device 10 sharpens the image (step S12). For example, in order to obtain a sharpened image, in equation (1),
J (x) = (I (x) -A) / t (x) + A ... (6)
Will be. Here, in order to prevent division by zero, a constant t0 is given, and the formula (6) is calculated by max (t (x), t0) instead of t (x).

In addition to the above He algorithm, various improved He methods, a method of maximizing the contrast of the image for removing fog, a method using a median filter, a method using a Markov random field model, etc. may be used, and the image is sharp. It is good if it is converted. The information processing server device 10 may remove fog by image processing using AI.

(3.3 Visibility estimation subroutine)
Next, the visibility estimation subroutine will be described with reference to FIGS. 16, 17A and 17B.
FIG. 16 is a flowchart showing a subroutine for estimating visibility. 17A and 17B are schematic views showing an example of visibility calculation.

As shown in FIG. 16, the information processing server device 10 calculates the light attenuation parameter (step S20). Specifically, the information processing server device 10 applies the relational expression (2) between the transmission map t (x) and the distance from the camera C to the image 30 which is I (x), and applies the light attenuation parameter beta. To calculate.

More specifically, since t (x) = 1-I (x) / A is obtained in the equation (5), t (x, y) for each two-dimensional pixel has a shading value in the entire image. It is a light source attenuation image.

From the bottom (front) of the image where x = 0 ((x, y) = (0,0)) to the top of the image 30 where x = MAX ((x, y) = (p, q)) When Sigma is the sum of all pixels up to The unknown number β is calculated so that is minimized.
E = Sigma | I (x) -exp (-beta x d (x)) | ^ 2> minimization ... (7)
This equation (7) can be obtained by the least squares method with the coefficient beta as an unknown number. For example, as a necessary condition, the condition that the first partial derivative of E by beta is 0 can be applied, various numerical solutions can be applied, and the simplest steepest descent method is applied, and the coefficients beta and d are applied. Find (x). The larger the light attenuation parameter beta, the more rapidly the change of fog etc. is compared with the distance in the depth direction. When applied to various images, 0.1 <beta <0.5.

As an example, the information processing server device 10 applies a relational expression between the transmission map and the distance from the photographing means to the image to calculate a light attenuation parameter indicating the degree of light attenuation. Function.

Next, the information processing server device 10 calculates the visibility from the depth direction distance estimated from the object or the depth direction distance calculated from the formula (3) based on the relational expression (2) of the light transmission map. (Step S21). Specifically, the information processing server device 10 determines the distance in the depth direction of each object estimated in step S6 based on the calculated exp (−beta × d (x)) according to the light attenuation parameter beta, or in step S7. Visibility is calculated from the calculated depth distance. In the case of the depth direction distance of each object estimated in step S6, more specifically, as shown in FIG. 17A, the control unit 16 determines I (x) in the partial image area of each object which is an example of a predetermined image area. ) Etc., the density distribution of the function exp (−beta × d (x)) in the value of the light attenuation parameter beta (for example, beta1, beta2, etc.) calculated in step S20 with respect to the value indicating the density of fog or the like. Apply. The value of the function exp (−beta × d (x)) and its reciprocal are examples of the value of light transmission based on the relational expression to which the light attenuation parameter is applied. Further, in FIG. 17A, in the case of the depth direction distance calculated in step S7, the horizontal axis is the depth direction distance dv of the equation (2), and the position of the pixel indicated by the vector v on the image plane P is in the predetermined image area. Correspond.

As an example of the visibility in which the value of light transmission based on the relational expression to which the calculated light attenuation parameter is applied becomes a predetermined value, as shown in FIG. 17A, the control unit 16 controls the distance at which the function value becomes a predetermined value (for example, , V1, v2) are the visibility. For example, the function value of the function exp (−beta × d (x)) on the right side of the equation (2) is an example of the light transmission value based on the relational expression to which the light attenuation parameter is applied. In the function exp (−beta × d (x)), d (x) may be a value of d1 or d2. A plurality of threshold values may be set according to the level of visibility.

As a value indicating the density of fog such as I (x) in the partial image area of each object, which is the value on the left side of the equation (2), for example, the equation (8) which is a modification of the equation (6)
t (x) = I (x) -A / (J (x) -A) ... (8)
It is calculated from. The pixel indicated by the vector x is a pixel corresponding to the vector v on the image plane P in the case of the depth direction distance calculated in step S7.

Further, the estimated value of t (x) may be a value calculated by any component of RGB in the patch in the image before sharpening of the image 30. For example, the estimated value of t (x) may be the minimum value of I / A in a certain color component in the partial image region, and may be a value obtained by subtracting the minimum value in the patch from 1.

As shown in FIG. 17B, the control unit 16 may apply the concentration distribution with reference to d1 which is closer to the camera C. Further, the control unit 16 may calculate the visibility by linearly approximating the distance between d1 and d2 with t (d1) and t (d2).

Since t (x), which is a light attenuation function, has an extreme value distribution from the front to the depth, the control unit 16 finds the maximum and minimum values, and the control unit 16 has a physical distance (in units of m) to each. May be assigned. For example, the control unit 16 integrates the brightness of the image 30 before it is sharpened for each horizontal line from the front to the depth direction, obtains the total brightness, and generates noise. In consideration, 10% may be the minimum value and 90% may be the maximum value with respect to the total value.

As described above, in the information processing server device 10, the visibility calculation means for calculating the visibility from the distance in the depth direction based on the relational expression to which the calculated light attenuation parameter is applied also functions as an example. The information processing server device 10 is a light transmission value based on the relational expression to which the calculated light attenuation parameter is applied, and is based on the light transmission value in the predetermined image region and the depth direction distance. A visibility calculation means for calculating visibility also functions as an example. The information processing server device 10 also functions as an example of a visibility calculation means for calculating the visibility at which the value of light transmission based on the relational expression to which the calculated light attenuation parameter is applied becomes a predetermined value.

As described above, according to the present embodiment, an image taken outdoors by the camera C is acquired, a sharpened image obtained by sharpening the image is generated, an object is extracted from the sharpened image, and the object is extracted from the pixel scale of the object. The physical scale of the object is converted, the distance in the depth direction of the object is calculated from the physical scale of the object, and the relational expression (2) between the light transmission map t (x) and the distance d (x) from the camera C is obtained. By applying it to an image, calculating the light attenuation parameter beta of the atmosphere, and calculating the field of view from the distance in the depth direction based on the relational expression (2) based on the calculated light attenuation parameter beta, no special measuring device is required. Then, the visibility can be estimated.

Further, since the object is extracted from the sharpened image, the image processing by AI becomes easy to apply, and the accuracy of visibility is improved.

Further, d (x) calculated by the relational expression (2) is only in pixel units, but physics in metric units or the like is performed by a table (object information DB12d) that converts the physical scale of an object from the pixel scale of an object. The scale can be obtained and the visibility can be calculated.

Further, when the object for calculating the depth direction distance cannot be recognized in the images 31 and 32, when the depth direction distance dv in the images 31 and 32 is calculated from the height of the installation of the camera C and the shooting direction, the road or the like. Visibility can be calculated even when the vehicle is not shown in.

Further, it is a light transmission value based on the relational expression (2) to which the calculated light attenuation parameter beta is applied, and is a light transmission value in an image region corresponding to an object which is an example of a predetermined image region, and an object. When calculating the visibility from the depth direction distance d (x) of, it is possible to apply a relational expression to the image and calculate the visibility corresponding to the depth direction distance d (x) of the object.

Further, when calculating the visibility in which the value of light transmission based on the relational expression (2) to which the calculated light attenuation parameter beta is applied becomes a predetermined value, the predetermined visibility corresponding to the predetermined value can be calculated.

Further, when the visibility is converted into the visibility level, the error in estimating the visibility of the physical distance can be absorbed, and the visibility can be expressed more stably depending on the level.

(Modification example)
Further, as shown in FIG. 18, when the road or runway is not horizontal, the information processing server device 10 may correct the value of dv according to the degree of inclination of the road or runway. For example, when the road is uphill, the road approaches the camera C, so that the value of dv is corrected to be shorter according to the inclination of the road. When the road is downhill, the camera C moves away from the road, so that the value of dv is corrected to be longer according to the inclination of the road. When the visibility is converted to the visibility level, the range of correction of the dv value may be absorbed by the visibility level even if the dv value is not corrected. Therefore, the information processing server device 10 is used. , It is not necessary to correct the value of dv in the equation (3) due to the inclination of the road or the like.

Next, a modified example of the transparency map t (x) will be described. The relational expression of the equation (8) is an extension of the light transmission map of the relational expression of the equation (2). Equation (8) is an equation in which in addition to the exponent of equation (2), each function indicating a sine wave, a chord wave, and a linear element is added.
t (x) = alpha1 x exp (-beta1 x d (x))
+ alpha2 x sin (-beta2 x d (x))
+ alpha3 x cos (-beta3 x d (x))
+ alpha4 × (beta4 × d (x))
... (9)

Here, for the weighting coefficients alpha1 to alpha4, for example, the default is 1. The weighting coefficients alpha1 to alpha4 may be changed depending on the image. The coefficients beta1 to beta4, which are examples of the light attenuation parameters, are the parameter beta1 for the exponential element, the parameter beta2 for the sine function element, the parameter beta3 for the cosine function element, and the parameter beta4 for the linear element, respectively.

Regarding the plurality of light attenuation parameters beta1 to beta4, for the image I (x) taken by the camera C, in the relational expression of the equation (7), instead of exp (−beta × d (x)), the equation (9) By applying the transmission map t (x) of the above, the information processing server device 10 calculates each light attenuation parameter beta1 to beta4 by the minimum square method with the coefficients beta1 to beta4 as unknowns.

For beta1 to beta4, the histogram distribution of the image I (x) taken by the camera C (horizontal axis 0 to 255 luminance gradation, vertical axis, number of pixels in one image corresponding to the luminance value) is obtained. Therefore, with respect to this histogram distribution and equation (9), the information processing server device 10 may calculate each light attenuation parameter beta1 to beta4 by the method of least squares.

The actual fog density is characterized by changing from moment to moment in time and space. On the other hand, the conventional monotonically decreasing exponential function (Equation (2)) assumes a uniform distribution of fog shades. Therefore, it is considered that the accuracy is lowered by the amount contrary to the non-uniformity of the fog. However, as shown in equation (9), the relational expression includes elements of at least one of the sine function and the cosine function in addition to the exponential function, and the light attenuation parameter is calculated by the light attenuation parameter calculation means. In addition, if the parameters for the elements of at least one of the sine function and the cosine function are included, the field of view can be calculated more accurately even if the shade of fog is not uniform.

Next, a modified example of image sharpening will be described.

In the case of an image with a significant amount of fog, there are problems that the image cannot be sharpened sufficiently and the visibility cannot be calculated accurately just by performing the sharpening process of the image in step S2 once. Therefore, when the information processing server device 10 has only one image, the image sharpening process in step S2 may be repeated several times. For example, the information processing server device 10 obtains a temporary sharpened image in step S12 in the first process, and returns this as an input image to step S10, which is repeated several times.

Further, the information processing server device 10 performs a sharpening process for removing fog by image processing using AI or the like, and estimates the first light attenuation parameter (including the parameter of the equation (9)) by the equation (7). Then, the sharpened image may be returned to step S10 as an input image, and the estimation of the second light attenuation parameter by the equation (7) may be repeated. From the first light attenuation parameter and the second light attenuation parameter (for example, from the average value of the first light attenuation parameter and the second light attenuation parameter), even if the information processing server device 10 calculates the visibility in step S21. Good.

Furthermore, the present invention is not limited to each of the above embodiments. Each of the above embodiments is an example, and any one having substantially the same configuration as the technical idea described in the claims of the present invention and exhibiting the same effect and effect may be used. It is included in the technical scope of the present invention.

1: Visibility estimation system 10: Information processing server device (visibility estimation device)
20: Terminal device 30: Image C: Camera (shooting means)

Claims (8)

An acquisition method for acquiring an image taken outdoors by a photography method, and
An estimation means for estimating ambient light and a light transmission map from the image,
An image sharpening means for generating a sharpening image obtained by sharpening the image based on the ambient light and the transmission map.
An object scale conversion means that recognizes an object from the sharpened image and converts the pixel scale of the object into the physical scale of the object based on the information about the actual scale of the recognized object.
Depth direction distance calculation means for calculating the depth direction distance of the object from the depth direction component of the physical scale of the object, and
A light attenuation parameter calculation means for calculating a light attenuation parameter indicating the degree of light attenuation by applying the relational expression between the transmission map and the distance from the photographing means to the image.
A visibility calculation means for calculating visibility from the depth direction distance based on the relational expression to which the calculated light attenuation parameters are applied.
A visibility estimation device characterized by being equipped with. In the visibility estimation device according to claim 1,
When the object for calculating the depth direction distance cannot be recognized in the image, the visibility estimation device is characterized in that the depth direction distance in the image is calculated from the height of the installation of the photographing means and the photographing direction. .. In the visibility estimation device according to claim 1 or 2.
The relational expression includes elements of at least one of a sine function and a cosine function in addition to the exponential function.
A visibility estimation device, characterized in that the light attenuation parameter includes a parameter for an element of at least one of the sine function and the cosine function calculated by the light attenuation parameter calculation means. In the visibility estimation device according to any one of claims 1 to 3,
The visibility calculation means is a light transmission value based on the relational expression to which the calculated light attenuation parameter is applied, and is a visibility from the light transmission value in a predetermined image region and the depth direction distance. A visibility estimation device characterized by calculating. In the visibility estimation device according to any one of claims 1 to 4.
A visibility estimation device, wherein the visibility calculation means calculates the visibility at which a value of light transmission based on the relational expression to which the calculated light attenuation parameter is applied becomes a predetermined value. In the visibility estimation device according to any one of claims 1 to 5.
A visibility estimation device further comprising a visibility level means for converting the visibility into a visibility level. The acquisition means is an acquisition step of acquiring an image taken outdoors by the photographing means, and
The estimation means includes an estimation step of estimating an ambient light and a light transmission map from the image, and an estimation step.
An image sharpening step in which the image sharpening means generates a sharpening image obtained by sharpening the image based on the ambient light and the transmission map.
An object scale conversion step in which the object scale conversion means recognizes an object from the sharpened image and converts the pixel scale of the object into the physical scale of the object based on the information on the actual scale of the recognized object. ,
A depth direction distance calculation step in which the depth direction distance calculation means calculates the depth direction distance of the object from the depth direction component of the physical scale of the object.
A light attenuation parameter calculation step in which the light attenuation parameter calculation means applies the relational expression between the transmission map and the distance from the photographing means to the image to calculate the light attenuation parameter indicating the degree of light attenuation.
A visibility calculation step in which the visibility calculation means calculates the visibility from the depth direction distance based on the relational expression to which the calculated light attenuation parameter is applied.
A visibility estimation method comprising. Computer,
Acquisition means for acquiring images taken outdoors by shooting means,
An estimation means for estimating ambient light and a light transmission map from the image,
An image sharpening means for generating a sharpening image obtained by sharpening the image based on the ambient light and the transmission map.
An object scale conversion means that recognizes an object from the sharpened image and converts the physical scale of the object from the pixel scale of the object based on the information about the actual scale of the recognized object.
An object depth direction distance calculation means for calculating the depth direction distance of the object from the depth direction component of the physical scale of the object.
A light attenuation parameter calculation means for calculating a light attenuation parameter indicating the degree of light attenuation by applying the relational expression between the transmission map and the distance from the photographing means to the image, and
A computer-readable record of a program for a visibility estimation device, which functions as a visibility calculation means for calculating visibility from the depth direction distance based on the relational expression to which the calculated light attenuation parameters are applied. Medium.

Images