JP2019096121A - Road image processing device, road image processing method, road image processing program, and recording medium - Google Patents

Abstract

To provide a road image processing device, road image processing method, road image processing program and recording medium that detect track cracks, using captured images by inexpensive cameras without requiring expensive dedicated instruments.SOLUTION: A road image processing device 100 processing road images has: a lane detection unit 112 that detects a lane area from a still image serving the road image, or a still image taken out from a motion image serving the road image; a bird's-eye view generation unit 113 that generates a bird's-eye view image of the lane area; a cross section extraction unit 114 that extracts a primary-dimensional pixel group in a road traverse direction in the bird's-eye view image; and a track determination unit 115 that determines presence ot absence of tracks of the pixel group.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technology for analyzing and processing a photographed image of a road to detect rutting defects and visualizing and displaying the situation.

  Conventionally, three indicators of crack and flatness and rutting have been used as indicators for detecting defects in road pavement such as asphalt. In particular, rutting refers to a defect in which the portion where the left and right wheels of the car repeatedly depress sinks and continues to dent in the traveling direction of the lane.

  Since rutting may cause problems such as taking off the steering wheel at high speed and causing an accident, or causing rainwater to accumulate in the rain and causing a large splash on the pedestrian, check the rutting. It is important to repair in a planned manner.

  In the past, expensive special-purpose vehicles called road surface quality measuring vehicles were often used for inspection with these index values. The road surface measurement vehicle is equipped with a dedicated measurement function that detects the unevenness in the cross direction of the road by performing laser irradiation, so that it is possible to accurately detect rutting.

  On the other hand, regarding crack problems, an inexpensive inspection method combining a video camera and an AI has also been proposed (Non-Patent Document 1).

  An image obtained by photographing a road with a still image camera or a video camera is naturally obtained as an image which looks small in distance based on perspective. From such an image, a technique for detecting an area of one lane to be determined as a pavement defect or for applying an inverse perspective to the area to generate an image viewed from directly above, Various methods have been proposed. For example, Non-Patent Document 2 discloses detailed procedures of both detection processing of a lane area and generation processing of an image in which the lane area is viewed from directly above. The “image viewed from directly above” is referred to as a bird's eye view or an ortho image.

"Birth! Infrastructure × AI industry map adult images and suspicious people do not miss, what depth learning is the next sight The infrastructure maintenance and management market that AI aims for (paving edition 2: NTT COMWARE), http: //itpro.nikkeibp. co.jp/atcl/column/17/051900200/052400007/ The date of publication of the website on June 6, 2017 "Development of Road White Line Positioning Method by In-vehicle Camera Image Processing", Junshiro Kanda, Shinya Taguchi, Shogo Yoneyama, IPSJ Reports Advanced Traffic System (ITS), 2006-ITS-024, pp. 55-61, 2006 .

  Conventionally, rutting detection requires expensive dedicated equipment such as laser irradiation and stereo imaging, and there has been a problem that detection can not be performed using an image captured by an inexpensive monocular video camera or the like.

  Also, conventionally, in order to visualize rutting detection status in an easy-to-understand manner, expensive dedicated equipment is required, and the appearance of rutting detection using an image captured with an inexpensive single-lens video camera etc. is visualized in an easy-to-understand manner There was a problem that it was impossible.

  The present invention has been made in view of the above circumstances, and it is an object of the present invention to detect rutting using an image taken by an inexpensive video camera without the need for expensive dedicated equipment. It is in.

  Another object of the present invention is to visualize the rutting detection status in an easy-to-understand manner by using an inexpensive camera-captured image without requiring an expensive dedicated device.

  The present invention is a road image processing apparatus that processes a road image, and includes a lane detection unit that detects a lane area from a still image that is the road image or a still image extracted from a moving image that is the road image; It has a generation unit that generates a bird's-eye view image of a region, an extraction unit that extracts a one-dimensional pixel group in a cross road direction in the bird's-eye view image, and a determination unit that determines presence or absence of a rut of the pixel group.

  The present invention is a road image processing apparatus for processing a road image, which is a lane detection unit for detecting a lane area from a still image which is the road image or a still image extracted from a moving image which is the road image; For each pixel of each of the extracted pixel groups, the generation unit for generating a bird's-eye view image of the area, the extraction unit for extracting a plurality of one-dimensional pixel groups in the cross road direction in the bird's eye view image The coordinate of the direction is set, the position of the pixel in the pixel group is set in the horizontal coordinate, the position of the pixel group in the road traveling direction is set in the vertical coordinate, and the set three-dimensional coordinate is set. And a drawing unit for drawing the road image in a three-dimensional manner.

  This invention is a road image processing method which processes a road image which a computer performs, Comprising: The lane detection step which detects a lane area | region from the still image which is the said road image, or the still image taken out from the moving image which is the said road image. Performing a generation step of generating a bird's-eye view image of the lane area, an extraction step of extracting a one-dimensional pixel group in the cross road direction in the bird's-eye view image, and a determination step determining presence or absence of a rut of the pixel group .

  This invention is a road image processing method which processes a road image which a computer performs, Comprising: The lane detection step which detects a lane area | region from the still image which is the said road image, or the still image taken out from the moving image which is the said road image. The generating step of generating a bird's-eye view image of the lane area, the extracting step of extracting a one-dimensional pixel group in the cross road direction in the bird's-eye view image, and the pixel value of each pixel of each extracted pixel group Is set to the coordinate in the height direction, the position of the pixel in the pixel group is set to the coordinate in the horizontal direction, the position in the road traveling direction of the pixel group is set to the coordinate in the vertical direction, And drawing the road image in a three-dimensional manner based on the coordinates of.

  The present invention is a road image processing program for causing a computer to function as the road image processing device.

  The present invention is a recording medium storing a road image processing program for causing a computer to function as the road image processing apparatus.

  According to the present invention, it is possible to detect rutting using images taken with an inexpensive video camera without the need for expensive dedicated equipment.

  Further, according to the present invention, it is possible to easily visualize the rutting detection status using an image captured by an inexpensive camera without requiring an expensive dedicated device.

1 shows the configuration of a road image processing apparatus. The example of the still picture which photographed the road is shown. 2 shows a processing flowchart. An example of a bird's eye view is shown. The example of the pixel group of the cross section regarding a road crossing direction is shown. The flowchart of a rut determination process is shown. The example of the derivative value of a cross-sectional pixel group is shown. 5 shows a flowchart of a one-sided rut determination process. An outline of one-sided rut determination processing using deep learning is shown. An example of mapping to a visualization array is shown.

  One of the embodiments of the present invention is shown below.

  FIG. 1 shows the configuration of a road image processing apparatus according to the present embodiment.

  The road image processing apparatus 100 has the following functions. That is, the storage medium 101, the screen 102, the still image acquisition unit 111, the lane detection unit 112, the bird's eye view generation unit 113, the cross section extraction unit 114, the rut determination unit 115, the aggregation unit 116, the drawing unit 117, and the drawing data storage unit 121 are provided. Do.

  The storage medium 101 is a means for inputting the photographed image into the road image processing apparatus 100. In the present embodiment, a file of a moving image captured while traveling with a commercially available video camera mounted on a vehicle is stored in an SD memory card or the like as the recording medium 101, and the SD memory card is stored in the road image processing apparatus 100. It will be attached and read. The screen 102 is an output device such as a display.

  The still image acquisition unit 111 extracts a still image from the moving image file of the road image recorded in the storage medium 101. The still image acquisition unit 111 may receive the still image itself instead of the moving image file. The lane detection unit 112 detects a lane area from the still image. The bird's eye view generation unit 113 generates a bird's eye view image of the lane area. The cross-section extraction unit 114 extracts one-dimensional pixel groups in the cross road direction in the bird's-eye view image.

  The ruth determination unit 115 determines the presence or absence of rutting (the presence or absence of ruth) of the extracted pixel group. The first wheel position extraction unit 1151 extracts a first pixel portion corresponding to the left wheel passing position and a second pixel portion corresponding to the right wheel passing position from the group of pixels, and a first pixel portion corresponding to the left wheel passing portion; And a detection unit 1152 that determines the presence or absence of rutting of the pixel portion and the second pixel portion. The aggregation unit 116 aggregates the determination results of each of the plurality of pixel groups, and determines the presence or absence of rutting of the entire plurality of pixel groups.

  The drawing unit 117 sets the pixel value of each pixel of each pixel group extracted by the cross-section extraction unit 114 as coordinates in the height direction, and sets the position of the pixel in the pixel group as horizontal coordinates. The position of the group of pixels in the road traveling direction is set as vertical coordinates, and the road image is drawn in three dimensions based on the set three-dimensional coordinates. The drawing data storage unit 121 stores each pixel group extracted by the cross-section extraction unit 114.

  Each of the functions 111 to 117 and 121 performs rutting determination and visualization of results by performing the operation described in detail below.

  FIG. 2 shows an example of a still image obtained by photographing a road. When the captured moving image stored in the storage medium 101 is paused to focus on each frame, each frame is an image such as the captured image 201 of FIG.

  In the present image, a road passing on the left side of one lane on one side is shown, and the center line 204 is shown by a dotted line. The left end of the lane is indicated by a white line 203, and the left side is a sidewalk. Reference numeral 205 denotes the end of the sidewalk, from which a house facing the road is photographed on the left. The lane 202 is an area between the center line 204 and the white line 203. Judgment of rutting is targeted at the area of the lane 202.

  The rutting is a depression of a wheel (tire) mark continuous in the traveling direction of the road, reference numeral 206 shows a rutting of the left wheel of the vehicle, and reference numeral 207 shows a rutting of the right wheel of the vehicle.

  Rutting appears in the form of visible shadows on the pavement of the road in the photographed image, due to the contact of the rays. When the appearance as in the reference numeral 206 or the reference numeral 207 is recognized on only one side or both sides, it is possible to determine that the photographed image has rutting.

  The rutting determination operation is realized by the operation described in detail below.

  FIG. 3 shows a flowchart of rutting determination processing.

  First, when the road image processing apparatus 100 starts processing (procedure 300), the still image acquisition unit 111 reads the file of the captured moving image from the storage medium 101, takes out the still image at regular intervals, and proceeds to the subsequent procedure. Send (procedure 301).

  Note that taking out still images at regular intervals as mentioned here means, for example, when moving images are shot at 30 frames per second, 30 still images per second are taken out and all of them are processed in the following procedure. It may be a target, or thinning may be performed so that only a predetermined number of 30 sheets are to be processed, or the latitude / longitude coordinates of the vehicle and the moving speed may be acquired from the vehicle by GPS, for example 3 Still images corresponding to positions advanced by a fixed distance interval such as every meter may be extracted and processed.

  Next, the lane detection unit 112 detects an area corresponding to the lane 202 from the still image to be processed (procedure 302). Furthermore, the bird's eye view generation unit 113 generates a bird's eye view, that is, an image seen from directly above, of the detected lane area in the image (step 303).

  As already mentioned, many techniques are conventionally proposed as the processing method of the procedure 302 and the procedure 303, for example, details of the processing method of the procedure 302 and the procedure 303 are disclosed in Non-Patent Document 2. By using these techniques, procedures 302 and 303 can be implemented.

  FIG. 4 shows an example of a bird's-eye view 401 obtained as a result of procedure 303. Reference numeral 401 denotes the entire obtained image, which corresponds to an image of a region corresponding to the lane 202 in FIG. 2 as viewed from directly above. Reference numeral 402 indicates the rutting of the left wheel corresponding to reference numeral 206 in FIG. Further, reference numeral 403 denotes rutting of the right wheel corresponding to reference numeral 207 in FIG.

  Next, the cross-section extraction unit 114 extracts, from the image 401 of the bird's-eye view, the pixel group of the cross-section in the cross road direction (procedure 304). Specifically, with respect to the bird's-eye view image 401, the cross section in the cross direction of the road is the code 410 or 411 or 412, and the process of picking up the pixels on this line corresponds to the procedure 304. As a matter of course, the pixel group of the cross section, which is the processing result, becomes a horizontal one-dimensional image (one-dimensional information).

  Here, the cross-sectional extraction unit 114 may store the extracted cross-sectional pixel group in the drawing data storage unit 121 (procedure 310), for use in processing to be described later.

  The extraction procedure of the cross-sectional pixel group performs extraction on one or more cross-sections with respect to one bird's-eye view image 401, and executes the procedure 305 described below on the cross-sectional pixel group of each extraction result Do.

  At this time, the extraction regarding the one or more cross sections may be performed, for example, for all the lines of all the pixels and all of them may be processed in the step 305, or extracted at predetermined intervals among the lines of the pixels. For example, a line of pixels on a bird's-eye view image corresponding to a position advanced by a predetermined distance interval such as every 0.5 meters may be extracted as a process target. It is also good.

  FIG. 5 shows an example of a cross-sectional pixel group in the cross road direction obtained as a result of the procedure 304. The horizontal axis in FIG. 5 represents the horizontal axis (position) of the one-dimensional image in the cross road direction (the horizontal direction in FIG. 4). Further, the vertical axis in FIG. 5 represents the pixel value of the pixel at each coordinate, and the larger the pixel value is, the graph as illustrated above.

  In the present embodiment, the luminance in a color image is used as the pixel value. The scope of the present invention is not limited to this, and any index value in the color expression of R, G, B or H, S, V, etc. may be used, or these may be calculated by a fixed formula. The value obtained as a result may be used as the pixel value.

  In the graph of the pixel group of the cross section illustrated in FIG. 5, the feature of the pixel value corresponding to the rutting image indicated by reference numerals 402 and 403 in FIG. 4 appears as the waveform change indicated by reference numerals 502 and 503, respectively. ing.

  Next, the rut determining unit 115 performs a subroutine processing to determine whether or not rutting is present with respect to the obtained pixel group of the cross section in the cross road direction (procedure 305). Details of the subroutine will be described later.

  Finally, the tabulating unit 116 determines, for each of the predetermined road sections, the determination result of the one or more rutting presence or absence obtained by applying the procedure 305 to each of the pixel groups of the one or more cross sections. An aggregation process is performed, and a final determination process is performed to determine whether or not rutting is present in the road section (procedure 306). Thus, the process ends (step 307).

  In the aggregation process, for example, a determination process with a threshold of 50%, ie, a majority process, may be performed on each determination result that rutting is present or absent, or a determination based on a threshold other than 50%. You may process. That is, when the pixel group of the cross section determined to have rutting presence is equal to or more than a predetermined threshold (ratio) in the whole, it is determined as rutting presence as a final determination result, while rutting presence is determined. When the pixel group of the determined cross section is less than a predetermined threshold value in the whole, it is determined that rutting is "none" as a final determination result.

  Alternatively, final determination may be performed by hypothesis rejection based on the statistical significance rate as the aggregation process. Alternatively, it may be an aggregation process in which the final determination result is also set to “present” when each rutting determination result is determined to be “present” continuously at a fixed number of times before and after.

  Also, if it is determined by the subroutine processing (procedure 305) to be described later whether or not rutting has been made for each of the left wheel and the right wheel, the results of those left and right are counted once (for example, when at least one has rutting) The counting process may be performed after the counting result is regarded as being present, or the counting process may be performed by regarding the left result and the right result as the determination results at different positions.

  In addition, the counting process may perform counting on the determination results in each cross section extracted in step 304 with respect to a single bird's-eye view image, or based on a plurality of still images extracted in step 301. A plurality of bird's-eye view images may be obtained, and a plurality of cross sections obtained from the plurality of bird's-eye view images may be totaled by integrating the determination results of the presence or absence of rutting in each cross section.

  In particular, according to the notification of the Ministry of Land, Infrastructure, Transport and Tourism, it is recommended to determine the presence or absence of rutting on sections such as 20 m or 100 m of the road. By doing, the advantage is obtained that the decision can be adapted to the notification.

  FIG. 6 shows a flowchart of the rut determination processing subroutine called in step 305.

  First, when the subroutine is started (procedure 600), the rut determining unit 115 calculates the differential value of the pixel group for the pixel group of the cross section (procedure 601). Here, since the calculation of the differential value is a process of calculating the difference between adjacent values as a set, specifically, for example, the pixel values of the adjacent pixels may be subtracted.

  FIG. 7 shows an example of the differential value of the obtained cross-sectional pixel group.

  Next, the rut determining unit 115 performs a wheel position extraction process to extract a subset of pixel groups at which the left wheel passes and the right wheel passes from the original differential value (procedure 602). ).

  Specifically, for example, when it is assumed that the pixel group of the cross section is a 1000 dot horizontal line image, the differential value is obtained as a set of 1000 dot equivalent values, so the center of the lane is 500 Coordinates near the dot. Taking into consideration that the vehicle line width is about 3 m to 3.5 m on a general road, the left wheel passes through, for example, a position near about 0 dots to 200 dots in general passenger cars and large vehicles. It can be assumed that the wheel of (1) passes approximately 800 dots to approximately 1000 dots, and so on. Thus, in the present embodiment, for example, the rut determining unit 115 extracts a set of differential values for 200 dots corresponding to the coordinates of each pair on the left and right.

  As described above, there is an advantage in principle in the concept of the processing in which the extraction processing is performed on a specific predetermined area in the lane width to be an analysis target in the following procedure. That is, rutting, which is the object of detection and visualization in the present invention, means that the pavement is deformed or worn away by repeated passing with general passenger cars and especially large vehicles repeatedly putting wheels on substantially the same position in the lane. It refers to the trouble that comes to interfere with safe and comfortable traffic.

  That is, it is inevitable from the viewpoint of rutting occurrence principle that many vehicles run on a statistically biased basis so as to step a specific position in a lane, and such a specific position is a fixed device in advance. Since this is a parameter that can be embedded as a constant, the mechanism for extracting and processing the pixel at a specific position in the lane with fixed parameters as described above is essential in view of the application purpose. It can be said that there is

  In the present embodiment, an example is shown in which the passing positions of the left and right wheels are fixedly set to 200 dots from the end point. The present invention can be applied to other embodiments as well, for example, the left wheel may have coordinates in the range of 100 dots to 250 dots, and on the basis of the fact that the lane width is different between the general road and the expressway. Different sets of settings may be prepared such as constant for constant and constant for freeway.

  Also, for example, 0% to 20% of the total number of dots in order to be able to process images captured with various resolutions instead of fixed width such as 1000 dots in width (pixel group of cross section) of the original image. It is good also as specifying by a ratio.

  Next, the rut determining unit 115 performs a subroutine process of determining whether or not rutting is present on one side with respect to the extraction result of the extracted left wheel position (procedure 603). Details of the subroutine will be described later.

  Next, the rut determining unit 115 similarly performs a subroutine process of determining whether there is rutting or not on one side with respect to the extraction result of the extracted right wheel position (procedure 604). Thus, the ruth determination processing subroutine is ended (step 605).

  In this embodiment, after the differential value of the pixel group is calculated in step 601, the process is performed to extract a subset corresponding to the left and right wheel positions in step 602. The scope of the present invention is not limited to this, and the order of the procedure 601 and the procedure 602 may be interchanged. This is because mathematically equivalent processing is one in which the one-dimensional set is differentiated and then the subset is extracted and the subset is extracted and then differentiated.

  FIG. 8 shows a flowchart of the one-sided rut determination process subroutine called in the steps 603 and 604.

  First, when the subroutine is started (Step 800), the rut determining unit 115 scans the one-dimensional set of differential values of the left or right wheels extracted in Step 602 of FIG. Perform determination processing. That is, first, it is detected that a positive value of the predetermined differential value or more (predetermined threshold value or more) appears (procedure 801). This corresponds to detecting the portion 701 in the example of the differential value shown in FIG. If it does not appear (in the case of no detection), the result is returned as a detection NG, that is, no rumor and the processing is ended (procedure 805).

  Next, when a positive value is detected in step 801, it is detected that a value (absolute value is equal to or less than a predetermined threshold) having a constant value close to zero within a certain range continues (step 802). This corresponds to the detection of the portion 702 in the example of the differential value shown in FIG. If it does not appear, the result is returned as a detection NG, that is, without a rumor, and the processing is finished (procedure 805).

  Next, when a series of values approximating to zero is detected in step 802, it is detected that a negative value of the differential value is equal to or less than a predetermined value (equal to or less than a predetermined threshold) appears (step 803). This corresponds to detecting the portion 703 in the example of the differential value shown in FIG. If it does not appear, the result is returned as a detection NG, that is, without a rumor, and the processing is finished (procedure 805). On the contrary, since all determinations from the step 801 to the step 803 are detection OK if they appear reversely, the result is returned as ruth and the processing is ended (step 804).

  In addition, the constant value (threshold value) which shows the said "constant" may be a value defined suitably. In the above-described procedure, it is detected whether or not the derivative value is a continuous change such as “positive value” → “zero” → “negative value”. This is, in other words, a continuous rise in which a sharp rise of the pixel value first occurs with respect to the original pixel value before differentiation, and then a sharp drop of the pixel value occurs at a position slightly apart. It is mathematically equivalent to the process of detecting a change.

  According to the above-described procedure, the road image processing apparatus 100 can detect a rutting defect (there is rutting) based on a moving image taken while driving while mounting a video camera on a vehicle. Since such detection does not require expensive equipment such as measurement equipment by laser irradiation or stereo imaging equipment for stereoscopic viewing, it is possible to realize rutting defect detection using general inexpensive equipment. You can get an advantage.

  The drawing unit 117 may display the detection result on the map, for example, as follows. That is, the drawing unit 117 displays the detection result as a color-coded line segment on the map of the plane corresponding to the moving image file recorded in the recording medium 101. Specifically, the section where rutting is detected is the road Visualization may be performed such as drawing a red line along the.

  Such display on a map is provided as a standard function in software such as Google Earth, so using the function, the detection result is input in a standard file format such as KML format input to the software. The output can realize the visualization.

  Such a display may be displayed on the screen 102 of the road image processing apparatus 100, or the road image processing apparatus 100 may output a file of the detection result and another file having a function of displaying only the file. It is good also as composition of inputting into a device and realizing the above-mentioned visualization.

  Generally, while image analysis takes a long time, it only needs to be performed once, and reading and displaying the file as a detection result on the screen is an operation that does not take much processing time but is repeatedly executed. Therefore, the configuration in which functions are shared as described above has practical advantages.

  In the present embodiment described above, the rut determination processing subroutine called in step 305 is based on the concept of detecting a waveform change by program processing (algorithm) based on the above-described flowchart and determining presence or absence of a rut. An embodiment has been shown. The present invention is also applicable to a method of forming a ruth determination processing subroutine using deep learning (DL), which is one of AI techniques, in addition to the above determination method.

  DL is a technology that makes a determination based on the idea of neural networks. When using the DL, the correct data group called teacher data is input to the learning function of the DL in advance in the product development and manufacturing stage to learn the correct data group according to the connection of the neural network previously held in the DL. Let it go and generate a trained model.

  As a result, the DL holding the learned model can return the determination result as to whether it is of the same type as the previously learned data by subsequently inputting the target data to the determination function of the DL. The advantage of DL is that human beings do not have to consider in advance the characteristics of the waveform etc. when determining the data to be targeted, but only a large number of correct data groups need to be learned in advance, ie at the product development / production stage. The point is that it is possible to judge the data with high accuracy. Because of this advantage, DL is widely used, and many available open source implementations are available, and DL is readily available.

  Below, the specific structural example (modification) of the road image processing apparatus using DL is shown.

  FIG. 9 shows an outline of the one-sided rut determination process using deep learning.

  In this configuration example, in the rut determination processing subroutine called in step 305, the one-side rut determination processing subroutine called in steps 603 and 604 is replaced with the execution shown above from the execution shown above. An example is shown.

  A flow chart 902 of FIG. 9B is a flow chart of the one-side rut determination processing subroutine called in the steps 603 and 604. The rut determining unit 115 incorporates DL software that holds a learned model.

  First, when the subroutine is started (procedure 920), the rut determining unit 115 inputs the one-dimensional set of the given differential values to the software of the DL holding the learned model, and there may or may not be rutting. Obtain a determination result (procedure 921). Next, the rut determination unit 115 returns the determination result and ends the processing (procedure 922).

  The point of this configuration example is that the software of the DL used in the above-described series of procedures is subjected in advance to a learning process in the development and manufacturing stages of the road image processing apparatus 100 to generate a learned model. .

  A flowchart 901 of FIG. 9A shows the prior learning process. As described above, the DL learns the connection of the neural network by inputting a large amount of teacher data to the learning function of the DL in advance, generates a learned model, and actually uses the DL as in the procedure 921. It is a technology that has a feature that makes it possible to determine with high accuracy whether the data is of the same type as the previously learned data, and its implementation can be widely used in open source and the like.

  In this configuration example, the same series of rutting determination procedures as described above, that is, the same processes as the procedures 301 to 305 in FIG. 3 and the procedures 601 to 602 in FIG. Also in the learning process performed in advance. However, at that time, the input to the procedure 301 is not a photographed image of a road where it is unclear whether there is rutting or not, and it is obtained by photographing a point where it is known that rutting actually exists. The captured image of the road is used as an input to the procedure 301.

  As a result, a result of the procedure 602 obtained by performing the series of processes from the procedure 301 to the procedure 602, that is, a subset regarding the pixel group where the left wheel passes and the right wheel passes is Since it is the above-mentioned set at a place where it is known that rutting is surely present, this is used as learning data of DL, and it is decided to use it as a learning data of DL. Further learning is performed (step 911).

  These series of procedures are repeated approximately several hundred times to several tens of thousands of times using various rutting images as input (procedure 912). Since the learning of the DL is sufficiently advanced by the above-described series of procedures, the learning process performed in advance in the development and manufacturing stages of the road image processing apparatus 100 is completed (procedure 913). At this point in time, the learned model held by the DL is a model in which the learning about the rutting features has been sufficiently advanced, so that the DL is used to construct the road image processing apparatus 100.

  As described above, when the road image processing apparatus 100 is actually used for rutting detection by following the procedure of preparing a large amount of teacher data for prior learning and causing the DL to learn, the above procedure With a simple procedure such as 921, it becomes possible to determine the presence or absence of rutting.

  Note that, as described above, many such DL implementations have been made public, and they provide a learning function and a determination function as an API (Application Programming Interface). Is possible.

  In the case of implementation using the above-described DL, there is an advantage that high determination accuracy can be obtained by preparing appropriate learning data.

  In addition, the above-mentioned "It repeats about several hundred times to several tens of thousands of times" means that the road image processing apparatus 100 is actually constructed using a DL that holds a learned model that has been actually learned to some extent, and an actual road image Let's analyze it, and if it does not get high enough, it can be realized by the procedure of giving additional learning.

  In the configuration example described above, the differential value of the pixel group is calculated in step 601, and a subset corresponding to the wheel position of the differential value is input to DL (learned model) to be learned, and It was decided to make a decision. The scope of the present invention is not limited to this, and the procedure 601 may be omitted, that is, not the differential value of the pixel value but the pixel value itself may be learned and input to the DL and determined. In this case, the rut determination unit 115 respectively generates a pixel (first pixel portion) of a portion corresponding to the left wheel passing position and a pixel (second pixel portion) of a portion corresponding to the right wheel passing position. By inputting to the learned model, it is determined whether or not each of the left and right ruts is present.

  Further, in the configuration example described above, the procedure 602 extracts a subset of the pixel group of the position where the left wheel passes and the position where the right wheel passes and extracts the left and right subsets into the DL (learned model) It was decided to input and learn, and to make DL judge. The scope of the present invention is not limited to this, and the procedure 601 and the procedure 602 are omitted, that is, not the left and right subsets but the original set of pixel groups are learned and input to the DL and the judgment input to the DL Implementation is also possible. In this case, the rut determining unit 115 determines the presence or absence of rumor by inputting the pixel group extracted by the cross-section extracting unit 114 to the learned model.

  In addition, it is also possible to omit only the step 602, that is, to carry out learning input of the differential value of the original pixel group to DL and judgment input to DL. In this case, the rut determination unit 115 determines the presence or absence of rumor by inputting the differential value of the pixel group extracted by the cross-section extraction unit 114 into the learned model.

  In addition to the above-described series of embodiments, it is also possible to provide a function to display three-dimensionally in accordance with the unevenness of the road regarding the rutting situation. A specific configuration example (modification) is shown below.

  In this configuration example, in the procedure 304 of FIG. 3, after extracting the pixel group of the cross section in the road crossing direction in the procedure 304 of FIG. 3, the pixel group of the cross section is further stored in the drawing data storage unit 121 (procedure 310). As described above, since the procedure 304 or the series of procedures from the procedure 301 to the procedure 304 is repeatedly performed with respect to a plurality of cross sections, the drawing data storage unit 121 Pixel groups will be saved.

  Since a pixel group of a single cross section is a one-dimensional data set having an extension in the horizontal axis direction, and a plurality of cross sections gather in the lane traveling direction, that is, the vertical axis direction, as a result, the drawing data storage unit 121 The pixel group of the cross section related to a plurality of places stored in is necessarily a two-dimensional data set on a plane.

  The drawing unit 117 performs drawing processing as follows using the stored data. The drawing unit 117 generates, for each point of the two-dimensional data set, a two-dimensional array for visualization in which the pixel value of the corresponding part is read as the height. That is, in the (X, Y) array, the value is an array having a height. The replacement may be, for example, simply pixel value = height.

  Since the array obtained by the above procedure is a set of (horizontal, vertical, height) points, for example, WebGL, three. Three-dimensional display can be achieved by using publicly available software having a 3D drawing function such as js.

  That is, for each pixel of each pixel group extracted by the cross-section extraction unit 114, the drawing unit 117 sets the pixel value as coordinates in the height direction, and the position of the pixel in the pixel group as horizontal coordinates. The position in the road traveling direction of the pixel group is set as the coordinate in the vertical direction, and the road image is drawn in three dimensions based on the set three-dimensional coordinate.

  FIG. 10 shows an example of mapping to the visualization array. Reference numeral 1001 denotes a visualization array to be displayed. Here, for example, the waveform of the pixel group of the cross section where rutting is present as illustrated in FIG. 5 is mapped to the array for visualization as, for example, 1020, 1021, or 1022, and the array is 3D. It shows how it is drawn.

  Reference numerals 1010, 1011 and 1012 indicate the direction of the cross section. For example, if a flat image without any unevenness is input, the mapping of, for example, the reference 1020 becomes almost parallel to the straight line of the reference 1010. .

  As indicated by reference numerals 1020, 1021 and 1022, the reason why a plurality of cross sections are arranged orthogonal to the traveling direction of the lane is the target of the display by storing the pixel group of the cross section related to a plurality of places as shown above. It shows how to

  If it is a problem that the pixel group in the cross section related to the plurality of places has a small total number, the interval in the vertical direction becomes a problem, linear interpolation (interpolation by the equation of Z = aY + b) or the like is performed. You may compensate the virtual vertex to the part that For example, reference numeral 1030 indicates the state of the above interpolation.

  According to the display procedure described above, the rutting can be displayed three-dimensionally in line with the unevenness of the road, and the rutting detection can be visualized in an easy-to-understand manner. That is, in this embodiment, the situation of rutting detection can be grasped at a glance, as compared with the case where the rutting detection result is displayed on a planar (two-dimensional) map.

  Further, in addition to the above display, for example, the rutting determination result itself may be superimposed and displayed, for example, a portion where it is determined that rutting is present has a red background.

  The display as described above may be displayed on the screen 102 of the road image processing apparatus 100, or the road image processing apparatus 100 may output a file of the processing result (the mapping example of FIG. 10). Only the file may be input to another device having a display function to realize the visualization. In particular, the road image processing apparatus 100 outputs the file of the processing result, inputs the file on the web server, and the web server transmits the information of the file to the web browser to draw the file on the web browser. It may be

  The present invention is not limited to the above embodiment, and various modifications are possible within the scope of the invention.

  In the present embodiment described above, the function of judging rutting with or without binary values is provided. By improving the above-described procedure, it is possible to calculate not only the presence or absence of rutting but also a value indicating the extent. For example, when the tabulating unit 116 tabulates one or more rutting determination results in step 306, it is determined that rutting is not only determined based on whether the threshold value is exceeded or not. A value indicating the degree of rutting may be output according to the number of individual determination results.

Alternatively, for example, with respect to a portion where a positive value of which the differential value detected in step 801 is a predetermined value or more appears, the rut determination unit 115 acquires a corresponding pixel value before differentiation, and rutting is performed according to the pixel value. A value indicating the degree may be output.

  The road image processing apparatus 100 described above includes, for example, a central processing unit (CPU), a memory, a storage (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device, and an input device. , An output device, and a general-purpose computer system can be used. In this computer system, each function of the road image processing apparatus 100 is realized by the CPU executing a predetermined program loaded on the memory. Further, the program for the road image processing apparatus 100 may be stored in a computer readable recording medium such as an HDD, an SSD, a USB memory, a CD-ROM, a DVD-ROM, an MO, or may be distributed via a network. it can.

100: Road image processing device
101: Storage medium
102: Screen
111: Still image acquisition unit
112: Lane detection unit
113: Bird's eye view generator
114: Cross section extraction unit
115: Ruth judgment unit
116: Summary section
117: Drawing unit
121: Drawing data storage unit

Claims (11)

A road image processing apparatus for processing a road image, comprising:
A lane detection unit that detects a lane area from the still image which is the road image or the still image extracted from the moving image which is the road image;
A generation unit that generates a bird's-eye view image of the lane area;
An extraction unit for extracting a one-dimensional pixel group in a cross road direction in the bird's eye view image;
A road image processing apparatus, comprising: a determination unit that determines presence or absence of a rut of the pixel group. The road image processing apparatus according to claim 1, wherein
The extraction unit extracts a plurality of the pixel groups.
The determination unit determines the presence or absence of each of the plurality of pixel groups.
A road image processing apparatus, comprising: a tabulation unit that tabulates determination results of each of the plurality of pixel groups and determines presence or absence of a rut of the plurality of pixel groups as a whole. The road image processing apparatus according to claim 1 or 2, wherein
The determination unit is
A wheel position extraction unit that extracts, from the group of pixels, a first pixel portion corresponding to a left wheel passing position and a second pixel portion corresponding to a right wheel passing position;
And a detection unit that determines that there is a rut when detecting a sharp rise and a sharp fall of the pixel value for each of the first pixel portion and the second pixel portion. Road image processing device. The road image processing apparatus according to claim 3, wherein
The detection unit is configured such that, with respect to differential values of the first pixel portion and the second pixel portion, the differential value becomes a plus value greater than or equal to a predetermined threshold value, and then becomes a minus value less than the predetermined threshold value. It is determined that there is a rut when it detects a road image processing apparatus. The road image processing apparatus according to claim 1, wherein
It has a learned model generated by learning, as teacher data, the pixel group extracted from the road image in which ruth exists, based on deep learning technology,
The road image processing apparatus, wherein the determination unit determines the presence or absence of a rut by inputting the pixel group to the learned model. The road image processing apparatus according to claim 1, wherein
It has a learned model generated by learning, as teacher data, the pixel group extracted from the road image in which ruth exists, based on deep learning technology,
The determination unit is
A wheel position extraction unit that extracts, from the group of pixels, a first pixel portion corresponding to a left wheel passing position and a second pixel portion corresponding to a right wheel passing position;
Providing a detection unit that determines presence or absence of the first pixel portion and the second pixel portion by inputting each of the first pixel portion and the second pixel portion to the learned model. Road image processing device characterized by A road image processing apparatus for processing a road image, comprising:
A lane detection unit that detects a lane area from the still image which is the road image or the still image extracted from the moving image which is the road image;
A generation unit that generates a bird's-eye view image of the lane area;
An extraction unit for extracting a plurality of one-dimensional pixel groups in the cross road direction in the bird's eye view image;
For each pixel of each of the extracted pixel groups, the pixel value is set to the coordinate in the height direction, and the position of the pixel in the pixel group is set to the coordinate in the lateral direction. A road image processing apparatus comprising: a drawing unit which sets a position to vertical coordinates and draws the road image in three dimensions based on the set three-dimensional coordinates. A road image processing method for processing a road image performed by a computer, comprising:
A lane detection step of detecting a lane area from the still image as the road image or the still image extracted from the moving image as the road image;
Generating a bird's-eye view image of the lane area;
Extracting a one-dimensional pixel group in a cross road direction in the bird's eye view image;
A road image processing method comprising the steps of: determining a presence or absence of a rut of the pixel group. A road image processing method for processing a road image performed by a computer, comprising:
A lane detection step of detecting a lane area from the still image as the road image or the still image extracted from the moving image as the road image;
Generating a bird's-eye view image of the lane area;
Extracting a one-dimensional pixel group in a cross road direction in the bird's eye view image;
For each pixel of each of the extracted pixel groups, the pixel value is set to the coordinate in the height direction, and the position of the pixel in the pixel group is set to the coordinate in the lateral direction. 7. A road image processing method comprising: setting a position to vertical coordinates; and drawing the road image in three dimensions based on the set three-dimensional coordinates.   A road image processing program for causing a computer to function as the road image processing device according to any one of claims 1 to 7.   A recording medium recording a road image processing program for causing a computer to function as the road image processing device according to any one of claims 1 to 7.

Images