JP6698191B1 - Road marking failure detection device, road marking failure detection method, and road marking failure detection program - Google Patents

Abstract

An object of the present invention is to evaluate the visibility of road markings more easily and more stably. A shape estimation unit 112 estimates the shape of a road marking from a captured image of a road to classify the presence or absence of the road marking and the type, and a density estimation unit 113 estimates the density of the road marking from the captured image. Then, the grade of blurring degree of the road marking is classified, the marking line evaluation unit 115 evaluates the blurring degree of the marking line for each section in which a plurality of photographed images are collected, and the output unit 116 determines the marking line for each section. Output the blurring degree. [Selection diagram] Figure 1

Description

  The present invention relates to a technique of analyzing a captured image of a road marking to detect a defect in the road marking.

  Conventionally, visual inspection work has been performed in order to maintain and manage road markings such as outside road lines and overtaking prohibition lines. The road marking is information for ensuring the safety of road users, and it is important to keep the visibility of the road marking in good condition. In determining the repainting of the road marking, an observer visually recognizes the road marking, and an index called a visual evaluation rank is often used to rank the road markings according to the impression of the visual recognition.

National Road Signs and Markers Association, "Road Marking Handbook", Published by National Road Signs and Markers Association, October 1, 2012

  Visual evaluation was conducted by humans in order to evaluate the visual evaluation rank. Since the evaluation depends on the impression of the observer, the evaluation may be different depending on the observer, and there is a problem that the evaluation cannot be uniquely determined by using a physical quantity or mechanical processing.

  Conventional visual inspection work only calculates the degree of blurring for each road marking. For road markings that have a shape that extends along a marking line or marking line-like road (hereinafter simply referred to as "marking lines"), we did not know the degree of blurring for a certain section of the road. There was a problem that it was difficult to judge whether or not to implement it for a certain section.

  The present invention has been made in view of the above, and it is an object of the present invention to more easily and more appropriately evaluate the visibility of road markings.

  A road marking defect detecting device according to the present invention is a road marking defect detecting device for detecting a defect of the road marking from a photographed image of a road marking including at least one of a road marking, a marking line and a non-legal display. A shape estimating means for estimating the shape of the road marking from a photographed image to classify the presence or absence and the type of the road marking, and estimating the shade of the road marking from the photographed image classified as having the road marking Then, the density estimation means for classifying the degree of blurring degree of the road marking, and when the road marking is a lane marking, a lane marking for evaluating the grading degree of the lane marking for each section in which a plurality of the photographed images are combined. It is characterized by having an evaluation means and an output means for outputting the blurring degree of the lane marking for each section.

  According to the present invention, the presence or absence of a road marking reflected in the image, the type, and the degree of blurring are determined for each section based on a captured image of a road taken by a video camera or the like, so that the visibility of the road marking can be simplified. , Can be evaluated more appropriately.

It is a functional block diagram which shows the structure of the road marking malfunction detection apparatus in this Embodiment. It is a figure which shows the example of the captured image which imaged the road. It is a figure which shows the example of the class which the convolutional neural network used by a shape estimation part classifies. It is a figure which shows the example of the class which the convolutional neural network used by a shape estimation part classifies. It is a figure which shows the example of the class which the convolutional neural network used by a gray level estimation part classifies. It is a flowchart of the main routine of the road marking defect detection device. It is a flowchart of a shape estimation subroutine. It is a figure which shows the example which acquires a partial image from the picked-up image which imaged the road. It is a figure which shows the example in which the same road marking straddles several partial images. It is a figure which shows the example which combined several partial image of FIG. 8 and set it as one partial image. It is a flowchart of a gradation estimation subroutine. It is a flowchart of a lane marking evaluation subroutine. It is a figure which shows the example of calculation of the score of a partial image. It is a figure which shows the example of calculation of the score of a partial image. It is a figure which shows the example in which one division line was detected in the some location in a picked-up image. It is a graph which shows the example which weights the score of a partial image. It is a figure which shows the graph which converts the average value of the score of the still image in an area into the level of an area. It is a flowchart of a result output subroutine. It is a figure which shows the example of the screen of the application using the output result of the road marking malfunction detection apparatus. It is a figure which shows the example of the screen of the application using the output result of the road marking malfunction detection apparatus.

  One of the embodiments of the present invention will be described below.

[Configuration of road marking image processing device]
FIG. 1 is a functional block diagram showing the configuration of the road marking image processing device according to the present embodiment.

  The road marking defect detection apparatus 100 shown in FIG. 1 includes a still image acquisition unit 111, a shape estimation unit 112 including a convolutional neural network 121 and a parameter 122, a density estimation unit 113 including a convolutional neural network 123 and a parameter 124, The imaging position abutting unit 114, the lane marking evaluation unit 115, and the output unit 116 are provided.

  The still image acquisition unit 111 acquires a captured image (still image) used to determine the degree of blurring of the road marking. In the present embodiment, a commercially available video camera is mounted on a vehicle, a file of a moving image captured while traveling is stored in a storage medium 101 such as a memory card, and the memory card is attached to the road marking defect detection device 100. And read it. The still image acquisition unit 111 acquires a still image from the captured moving image read from the storage medium 101 at regular intervals. When focusing on each frame of the captured moving image stored in the storage medium 101, each frame can be regarded as a still image. Note that this device does not depend on the method of capturing a still image or a moving image, and may use a still image obtained by taking a close-up of the road marking using a commercially available camera on the ground, for example. The still image acquisition unit 111 may acquire a captured moving image or a still image via a network.

  FIG. 2 is a diagram showing an example of a photographed image of a road. A photographed image 201 shown in the figure is a photographed image of a road on the left side of one lane on each side. In the photographed image 201, a pedestrian crossing notice 202, a road outside line 203, a right-hand side prohibition passage 204 for overtaking, and a sidewalk edge 205 are reflected. On the left side of the edge 205 of the sidewalk, houses and the like facing the road are reflected. The photographed image 201 may have any width and height, but in the present embodiment, the photographed image 201 has a width of 3600 pixels and a height of 2000 pixels.

  The shape estimation unit 112 classifies the types of road markings shown in the still image. In the present embodiment, the shape estimation unit 112 classifies the types of road markings captured in a still image using a convolutional neural network.

  The convolutional neural network is a technique for classifying images based on the concept of neural networks. In a convolutional neural network, images are classified in units called classes. For example, a convolutional neural network capable of classifying apple images and mandarin orange images can be said to be a network having two classes, an apple class and a mandarin orange class. When using a convolutional neural network, at the product development and manufacturing stage, prepare correct answer images called teacher images for each class you want to classify, and input the correct answer images to the learning function of the convolutional neural network. A group of correct images is trained on the connection of the neural networks held inside the neural network, and the learning result is set as a parameter of the convolutional neural network. Thereby, the convolutional neural network can return the result of classifying the target image into the class learned in advance by inputting the target image to the determination function of the convolutional neural network. At this time, the convolutional neural network returns the result as a sequence of certainty factors in which the target image is each class, such as "80% certainty factor that is an apple and 20% certainty factor that is a mandarin orange". be able to. The advantage of the convolutional neural network is that it is not necessary for humans to consider the characteristics of the image in advance when determining the target image, and a large amount of correct image groups can be learned in advance, that is, at the product development and manufacturing stage. The point is that the target image can be determined with high accuracy. Because of this advantage, convolutional neural networks are widely used, there are many open source libraries available, and convolutional neural networks are readily available. This device realizes a learning function and a judgment function by using an open source library for the convolutional neural networks 121 and 123.

  3A and 3B are diagrams showing examples of classes classified by the convolutional neural network 121 used in the shape estimation unit 112.

  In the present embodiment, a correct image group is prepared for each class shown in FIGS. 3A and 3B in advance, that is, in the development and manufacturing stage of the present device, and the correct image group is used for the learning function of the convolutional neural network 121 used by the shape estimation unit 112. Is input and the learning result is set as the parameter 122 of the convolutional neural network 121. When the apparatus is in operation, a target image is input to the determination function of the convolutional neural network 121 to determine whether or not the road marking is reflected in the target image, and to classify the reflected road marking type.

  Road markings include road markings, lane markings, and non-statutory markings. Further, there are a plurality of types of road markings. The type of road marking to be inspected differs depending on whether the inspection body is a road administrator or a prefectural public safety commission.

  In the present embodiment, the nine types of road marking classes shown in FIG. 3A and the classes of objects other than the three types of road marking shown in FIG. 3B are included in the classes classified by the convolutional neural network 121. Specifically, as shown in FIG. 3A, a pedestrian crossing notice 311, a stop line 312, a pedestrian crossing 313, a right-hand side portion forbidden to pass, 314, a stop character 315, a road outside line 316, a T-shaped intersection cross mark 317, The type 10 intersection cross mark 318, the road surface 320 not showing the road marking, the wall surface 331, the vehicle body 332, and the sidewalk 333 shown in FIG. 3B are included in the class classified by the convolutional neural network 121. Therefore, the convolutional neural network 121 has a total of 12 classes. The number of classes to be classified may be set arbitrarily. The road surface 320 in which the road marking is not reflected is a class for determining the presence or absence of the road marking. The three types of classes of the wall surface 331, the vehicle body 332, and the sidewalk 333 are classes in which the reflection of objects other than the road marking is classified.

  When preparing the correct answer image group to be input to the learning function, prepare a still image group in which the classification target is reflected, and cut out the portion of the still image group in which the classification target of each class is reflected using commercially available image processing software. The correct answer image. This work is performed by a person as part of the manufacturing work at the development and manufacturing stage of this device. In the present embodiment, a still image group in which a road is photographed is prepared for a correct image, and a portion of the still image group in which a classification target is reflected is cut out with commercially available image processing software. At least 1000 correct images are used for each class. The correct image used for learning may be square or rectangular as long as it is rectangular. For example, the stop line 312 may be prepared as a horizontally long rectangular image according to its shape.

  For classes other than the road marking class to be detected, if it is difficult to secure the number of correct images, a plurality of classes may be combined into one class. For example, when the T-type intersection cross mark 317 and the 10-type intersection cross mark 318 are road markings that are not the detection targets, the T-type intersection cross mark 317 and the 10-type intersection cross mark 318 are combined into one intersection cross mark class. Alternatively, the wall surface 331, the vehicle body 332, and the sidewalk 333, which are the classes of objects other than the road marking, may be combined and used as the class of the reflection of the object.

  The gradation estimation unit 113 classifies the blurring degree of the road markings reflected in the still image. In the present embodiment, the density estimation unit 113 classifies the blurring degree of the road markings reflected in the still image using a convolutional neural network. The degree of blurring refers to the degree of unclearness and loss of the coating film due to wear, peeling and deterioration of road markings.

  FIG. 4 is a diagram showing an example of classes classified by the convolutional neural network 123 used in the density estimation unit 113. In the present embodiment, a correct image group is prepared for each class shown in FIG. 4 in advance, that is, at the development and manufacturing stage of this device, and the correct image group is input to the learning function of the convolutional neural network 123 used by the gray level estimating unit 113. , The learning result is set as the parameter 124 of the convolutional neural network 123. When the apparatus is in operation, the target image is input to the determination function of the convolutional neural network 123 to classify the blurring degree of the reflected road marking.

  The number of stages to classify the degree of blurring depends on the inspection criteria of the inspection body. In the present embodiment, the degree of blurring is classified into three levels of level 1, level 2 and level 3, but the number of levels of blurring degree may be set arbitrarily. For example, it may be classified into 5 levels according to the number of visual evaluation ranks introduced in the road marking handbook (see Non-Patent Document 1).

  The definition of each level of blurring depends on the inspection standard of the inspection body. In the present embodiment, level 1 maintains the shape of the whole road marking, and the area of the unclear spot due to blurring occupies less than 30% of the whole road marking, and level 2 maintains the shape of the entire road marking. There is road marking that occupies 30% or more and less than 60% of the entire road marking due to blurring. Level 3 does not maintain the overall shape of the road marking due to loss, or the area of the blurry area due to blurring is entirely. Road markings that account for 60% or more of

  Each of the classes 401 to 403 in FIG. 4 corresponds to blurring level 1 to 3. The class 404 of the road surface and the object in which the road marking is not reflected is composed of four types of a road surface 441, a wall surface 442, a vehicle body 443, and a sidewalk 444.

  The convolutional neural network 123 used by the density estimation unit 113 is composed of classes 401 to 403 of blurring level 1, level 2 and level 3 and a class 404 of a road surface and an object in which no road marking is shown. A parameter 124 in which a class of a type is learned in advance is set.

  The image used for learning may be square or rectangular as long as it is rectangular. Further, the correct image group used for learning at each level may be configured by a single type of road marking, or may be configured by a plurality of types of road marking, and may be used for learning the convolutional neural network 121. The correct answer image may be reused.

  In the present embodiment, the correct image used for learning the convolutional neural network 121 is reused. Specifically, for the correct images of the three classes of level 1, level 2, and level 3, the class that is the road marking, that is, the pedestrian crossing notice 311, the stop line 312, the pedestrian crossing 313, the right side portion for overtaking Correct answer images of the sticking out prohibition 314, the stop character 315, the road outside line 316, the T-shaped intersection cross mark 317, and the ten-shaped intersection cross mark 318 are classified for each blurring level and reused. As for the correct answer image of the road surface and the object in which the road marking is not reflected, the correct answer images of the road surface 320, the wall surface 331, the vehicle body 332, and the sidewalk 333 in which the road marking is not reflected are reused as they are. The standard number of images used for learning varies depending on the library used, but in the present embodiment, at least 1000 or more images are used for each class.

  The shooting position matching unit 114 specifies the shooting point of the still image acquired by the still image acquisition unit 111 based on the position information of the shooting moving image. In the present embodiment, a commercially available GPS logger is mounted on a vehicle, a file of position information acquired while traveling is stored in a storage medium 102 such as a memory card, and the memory card is attached to the road marking defect detection device 100. And read it. Note that the present apparatus may use, for example, the information of the shooting position recorded in Exif in the shot still image, regardless of the method of recording the shooting position. The shooting position matching unit 114 may acquire a file of position information via a network.

  The lane marking evaluation unit 115 evaluates the blurring degree of the lane markings for a section in which continuous still images are collected in units of a fixed number of sheets. The lane markings that the lane marking evaluating unit 115 evaluates for each section are, for example, road markings classified into classes of a right-hand side protruding prohibition 314 and a road outside line 316 shown in FIG. 3A. In the present embodiment, continuous still images are grouped in units of six. If the still image acquisition unit 111 acquires still images at intervals of 3.3 m, the marking line evaluation unit 115 will evaluate the degree of blurring of the marking lines for each section of about 20 m. The length of the section, the interval at which the still images are acquired, and the number of still images collected as the section may be set arbitrarily.

  The output unit 116 outputs the identifier of the still image in which the road marking to be detected is detected, the position of the road marking in the still image, the type, and the blurring degree information. The frame number of the captured moving image from which the still image is acquired may be used as the identifier of the still image. The output unit 116 may output information on the shooting position of the still image in which the road marking is detected.

  The output unit 116 also outputs the blurring degree of the lane markings for each section.

[Operation of road marking image processing device]
Next, the operation of the road marking image processing device according to the present embodiment will be described. FIG. 5 is a flowchart of the main routine of the road marking defect detection device 100.

  First, when the road marking defect detection apparatus 100 starts processing, the still image acquisition unit 111 reads a file of a captured moving image from the storage medium 101, extracts still images at regular intervals, and extracts the extracted still image group from now on. (Step 501). The latitude/longitude coordinates and the moving speed of the vehicle are acquired by the GPS, and a still image corresponding to a position advanced by a constant distance interval of 3.3 m is extracted and processed.

  Next, the shape estimation unit 112 extracts a partial image including the road marking from each still image of the still image group by the processing of the shape estimation subroutine described later, and classifies the type of the road marking included in the partial image (procedure). 502). The partial image is an image obtained by cutting out a part of the still image, and is a processing unit for classifying the types of road markings and estimating the shading of the road markings. A plurality of partial images may be extracted from the still image.

  Next, the density estimation unit 113 classifies the blurring degree of the road marking included in the partial image by the processing of a density estimation subroutine described later (step 503).

  Next, the shooting position matching unit 114 reads the position information file from the storage medium 102 and specifies the shooting point of each still image in the still image group (step 504). When specifying the shooting point of a still image, for example, even when the shooting point is specified by acquiring the time when the frame corresponding to the still image extracted from the moving image was acquired and collating with the time when the position information was recorded. Alternatively, if the two times do not match, the position information before and after may be used for the estimation by performing a numerical calculation such as linear interpolation. Alternatively, if the data recorded in the storage medium 101 is not a moving image but a still image, the position information recorded in the Exif of the still image may be used.

  Next, the lane marking evaluation unit 115 divides the still image group into fixed sections by the processing of a lane marking evaluation subroutine described later, and evaluates the degree of blurring of the lane markings for each section (procedure 505).

  Next, the output unit 116 outputs the information of the position, type, and blurring degree in the still image of the road marking of the detection target reflected in each still image by the processing of the result output subroutine described later ( Step 506). The output unit 116 outputs the evaluation for each section for the lane markings.

  Through the above processing, it is possible to obtain information about the presence or absence of the road marking to be detected, the position in the still image, the type, and the degree of blurring from the still image or moving image of the road. Regarding the lane markings, it is possible to obtain information on the degree of blurring for each section.

(Shape estimation process)
FIG. 6 is a flowchart of the shape estimation subroutine.

  When the shape estimation subroutine is called, the shape estimation unit 112 divides the still image to be processed into rectangles, and among the respective portions (partial images) of the still image obtained by the division, the portions that meet the conditions described later. The partial image group consisting of images is sent to the subsequent procedure (procedure 601).

  FIG. 7 is a diagram showing an example of obtaining a partial image from the captured image 201. A rectangle 701 is a square rectangle having a height of 800 pixels and a width of 800 pixels, and the processed image starts from the point where the upper left apex of the rectangle is aligned with the captured image 201, and the rectangle is shifted by 400 pixels in the horizontal direction and 400 pixels in the vertical direction. It is divided without. Each part represented by a 2×2 rectangle in the captured image 201 is a partial image.

  It should be noted that the size of the partial image is arbitrary, but in view of the size of the road marking reflected in the still image, a size is ensured so that most of the road marking fits within the partial image, and It is rational to keep the size of other road markings and objects in the vicinity so that they do not enter as much as possible in view of the nature of the road markings and objects reflected on the still image of the road. Further, each side of the partial image may have a fixed length as in the present embodiment, or a plurality of types of partial images having different widths and heights may be used. For example, considering the perspective method, the partial image may be made smaller toward the upper side of the still image, that is, toward the back of the road.

  Since the partial image is acquired by dividing the entire still image, the partial image in the upper part of the still image includes a region that does not include the road surface or a region in which the road marking is small due to the perspective method. The partial image including such an area does not have the road marking, or even if it does, the size of the partial image is insufficient for estimating the shape and shading. Particularly in the latter case, it may cause a wrong estimation result to be output, and it is not preferable to send it to the subsequent procedure.

  In the present embodiment, considering that most of the road is captured in the lower area of the still image by the perspective method, the partial image of the lower portion of the still image is sent to the subsequent procedure. Specifically, the set of partial images that are wholly contained in the lower 60% area 702 of the captured image 201 in FIG. 7 is sent to the subsequent procedure. The conditions of the partial image to be processed are not limited to the above, and for example, the periphery of the still image may be excluded from the processing target, or the left and right edges of the still image may be excluded from the processing target.

  As a method of obtaining the partial image, the method of the present embodiment described above may be used, or a method of extracting using a known object candidate detection algorithm such as Selective Search may be used. Alternatively, a detection method using an existing neural network such as Region Proposal Network (RPN) in Faster R-CNN may be used.

  Next, the shape estimation unit 112 inputs the partial image group to the determination function of the convolutional neural network 121, and classifies each partial image into one of the classes shown in FIGS. 3A and 3B (step 602). Hereinafter, this classification result will be referred to as a shape classification result.

  Next, the shape estimation unit 112 determines, for each partial image included in the partial image group, whether the shape classification result is road marking (step 603). The partial image whose shape classification result is road marking is the processing target of the subsequent procedure (procedure 604), and the partial image whose shape classification result is not road marking is not the processing target of the subsequent procedure (procedure 605). That is, only the partial images classified into the class shown in FIG. 3A (excluding the road surface 320 on which the road marking is not reflected) are processed in the subsequent steps. The processing from step 603 to step 605 is repeated for each partial image included in the partial image group.

  Next, the shape estimation unit 112 combines a plurality of partial images into one partial image when the same road marking extends over a plurality of partial images, and combines the partial images in place of the plurality of partial images before the combination. The partial image is sent to the subsequent procedure (procedure 606). The same means that the shape classification results are in the same class. Spanning means a state in which, when two partial images are arbitrarily selected, both partial images are in the same class, and at least one set of sides forming both partial images intersect or overlap each other. Combining into one partial image means obtaining a partial image with the smallest rectangle that includes all of a plurality of partial images.

  FIG. 8 is a diagram showing an example in which the same road marking (preliminary notice 202 of a pedestrian crossing) straddles a plurality of partial images.

  In the partial images 801 to 804 of FIGS. 8A to 8D, the same pedestrian notice 202 is detected. Further, the partial images 801 to 804 are present at positions crossing each other. Therefore, the shape estimation unit 112 combines the partial images 801 to 804 to obtain one partial image 901 as shown in FIG. In addition to the pedestrian crossing notice 202, the photographed image 201 includes a roadway outside line 203 and a road surface marking of a right-hand side protruding forbidden 204 for overtaking. Even when these road markings extend over a plurality of partial images, they are likewise combined.

  Next, the shape estimation unit 112 outputs the partial image group and the shape classification result (procedure 607), and ends the shape estimation subroutine.

(Tint estimation processing)
FIG. 10 is a flowchart of the gradation estimation subroutine.

  When the density estimation sub-routine is called, the density estimation unit 113 inputs the partial image group output by the shape estimation unit 112 to the determination function of the convolutional neural network 123, and assigns each partial image to one of the classes shown in FIG. Classify (procedure 1001). Specifically, the grayscale estimation unit 113 classifies each partial image into a class of blurring levels 1 to 3 or a class of a road surface and an object in which the road marking is not reflected. In the present embodiment, the blurring level of the class of the road surface and the object in which the road marking is not reflected is set to level 0. That is, the gradation estimation unit 113 classifies the blurring degree of each partial image into four levels of levels 0 to 3.

  The gradation estimation unit 113 outputs a gradation classification result which is a classification result for each partial image (procedure 1002), and ends the gradation estimation subroutine. The grayscale classification result of each partial image includes a certainty factor for each of levels 0 to 3. The certainty factor is a numerical value of 0 or more and 1 or less indicating the possibility that the blurring degree of the partial image is at that level. The sum of the certainty factors of each level 0 to 3 is 1. For example, if a certain partial image has a level 0 certainty factor of 0, a level 1 certainty factor of 0.1, a level 2 certainty factor of 0.7, and a level 3 certainty factor of 0.2, the partial image The degree of blurring is likely to be level 2.

(Parcel line evaluation process)
FIG. 11 is a flowchart of the lane marking evaluation subroutine.

  When the lane marking evaluation subroutine is called, the lane marking evaluation unit 115, for each section in which continuous still images are collected in units of a fixed number of sheets, the shape classification result of the partial image in which the lane marking is detected in the still images in the section. The grayscale classification result is sent to the subsequent steps (step 1101). In the present embodiment, a section is composed of 6 continuous still images, and the shape classification result and the gradation classification result of the partial image in which the lane markings are detected in the 6 still images are sent to the subsequent procedure. The subsequent procedure is performed for each section to be processed.

  Next, the marking line evaluation unit 115 determines whether a marking line is detected in any of the still images in the section (step 1102). Specifically, the lane marking evaluation unit 115 determines whether or not a still image in the processing target section includes a right-side protruding portion prohibition 314 or a partial image of the road outside line 316 for passing the shape classification result. judge.

  When there is no partial image of which the shape classification result is the lane marking in the still image in the section, the lane marking evaluating unit 115 determines that there is no lane marking in the section (step 1103). If there is a section to be processed next, the lane marking evaluation unit 115 repeats the processing from step 1101 with the next section as the processing target (step 1107).

When the still image in the section includes a partial image whose shape classification result is the lane marking, the lane marking evaluation unit 115 calculates the score of the partial image based on the gray scale classification result of the partial image (step 1104). Specifically, the lane marking evaluation unit 115 uses the following equation (1) to calculate an expected value at the confidence levels confidence _m (m=1 to 3) of levels 1 to 3 of the grayscale classification result, and the partial image Compute the score score _subframe of.

  The partial image with the highest certainty factor of level 0 is regarded as an error in the shape estimation unit 112 and is not included in the score calculation of the partial image. In addition, the confidence levels of Levels 1 to 3 excluding the confidence level of Level 0 are used to calculate the score of the partial image. When estimating the shading of a partial image, the shading estimation unit 113 may be confused about Level 0 where there is no road marking and Level 3 where the road marking is blurred and difficult to see. In this case, if the level 0 is included in the calculation of the score of the partial image, the score of the partial image that should originally be close to the level 3 is underestimated. Therefore, the lane marking evaluation unit 115 calculates the score of the partial image using the certainty factors of levels 1 to 3 excluding the certainty factor of level 0.

  Calculation of the score of the partial image will be described using a specific example with reference to FIGS. 12A and 12B. The grayscale classification result of the partial image shown in FIG. 12A shows that the confidence level at level 0 is 0, the confidence level at level 1 is 0.1, the confidence level at level 2 is 0.7, and the confidence level at level 3 is 0.2. is there. Substituting the numerical values of levels 1 to 3 excluding the certainty factor of level 0 into the equation (1), the score of the partial image=(1×0.1+2×0.7+3×0.2)÷(0.1+0.7+0 .2)=2.1.

  The grayscale classification result of the partial image shown in FIG. 12B has a level 0 certainty factor of 0.4, a level 1 certainty factor 0, a level 2 certainty factor 0, and a level 3 certainty factor 0.6. Substituting the numerical values of levels 1 to 3 excluding the certainty factor of level 0 into the equation (1), the partial image score=(1×0+2×0+3×0.6)÷(0+0+0.6)=3. As shown in FIG. 12B, the score of the partial image is evaluated as 3 more appropriately even in the case where the shade estimation unit 113 gets lost at level 0 or level 3.

  When a partial image is a combination of a plurality of partial images into one partial image, a gray level classification result is obtained for each of the partial images before combination, and an expression ( The score may be obtained in 1), and the average of the obtained scores may be calculated as the score of the combined partial images.

Next, the lane marking evaluation unit 115 calculates the score of the still image based on the score of the partial image (step 1105). In the present embodiment, the higher the degree of blurring of the division line in the still image, the higher the weight of the still image is evaluated, and the still image score is calculated. Specifically, the lane marking evaluation unit 115 uses the following equation (2) to power the score sub- _framek of the partial images in one still image to the power of the index γ and weights the average to obtain the still image. Calculate the score _frame of.

Here, K is the number of partial images in which the lane markings estimated to be the same in the still image are detected. The lane markings estimated to be the same are lane markings that can be considered to be the same based on the detected position of the lane markings on the still image. For example, the lane markings detected as a plurality of partial images at positions along the road extension on the still image can be estimated to be the same. It is possible to presume that the lane markings detected as a plurality of partial images at positions apart from each other on the left and right of the still image without following the road extension are lane markings different from each other. For a still image with K=0, that is, a still image in which no dividing line is detected, the score of the still image is score _frame =1. That is, 1≦score _frame ≦3 ^γ .

  The advantages of the above calculation method will be described. For example, as shown in FIG. 13, it is assumed that one division line is detected as a plurality of partial images 1301 and 1302, and the scores of the partial images are 3 and 1, respectively. In this case, if the score of the still image is obtained by simply averaging the scores of the partial images, the score of the still image becomes 2. Therefore, the level of the lane line with a high degree of blurring to be evaluated as level 3 is lowered. There is a problem of evaluation.

  Therefore, in the present embodiment, after calculating the score of the partial image by performing weighting according to the blurring degree of the division line in the partial image, the score of the partial image is aggregated to calculate the score of the still image. And Specifically, the score of the partial image is raised to the power of the index γ to convert the scale so that the score of the partial image increases as the degree of blurring of the dividing line of the partial image increases, as shown in the graph of FIG. The score of the partial image was weighted so as to be functionally high. The value of the index γ is arbitrary, but in the present embodiment, the index γ=log60/log3. The weighting method is not limited to the above. It suffices that the higher the degree of blurring of the marking line, the higher the weighting.

  The lane marking evaluation unit 115 repeats the processing of steps 1104 and 1105 for each still image in the section, and calculates a still image score based on the blurring degree of the lane marking for each still image in the section.

  If different lane markings appear in the still image, the score of the still image is calculated for each lane marking. For example, in the photographed image 201 of FIG. 2, the outside line 203 of the roadway and the right-hand side protruding portion 204 for overtaking are shown. For a section including the photographed image 201, a still image score based on the partial image in which the outside road 203 is detected and a still image score based on the partial image in which the right-side protruding portion 204 for overtaking is detected are obtained. .. In the following procedure 1106, the degree of blurring of the road outside line 203 and the degree of blurring of the right-hand side protruding prohibition 204 for overtaking are obtained.

Next, the lane marking evaluation unit 115 totals the scores of the still images in the section, and obtains the blurring degree of the lane markings in the section based on the totalized result (step 1106). Specifically, the lane marking evaluation unit 115 uses the following expression (3) to calculate the score score _section of the section based on the average of the score score _frames of the still images in the _section .

Here, N is the number of still images in the section. const is an arbitrary constant in the range of 1 <const ≦ 3 ^γ. In this embodiment, const=2.5 ^γ . The score score _section of the _section can be calculated in the same manner by using the total value instead of the average value of the score _frame of the still image.

  In the present embodiment, in step 1105, the score is multiplied by γ, then the average value is obtained, and the result is inversely calculated and used in the next step 1106 without performing calculation to restore the original scale. , When the scores of still images in a section are totaled to determine the degree of blurring of lane markings in the section, the higher the degree of blurring, the higher the weighted evaluation and the totaled result. The present invention is not limited to this. For example, the average value obtained in the step 1105 may be raised to the power of 1/γ to return to the original scale, and then in the step 1106, the value may be raised to the power γ again and then aggregated. Alternatively, in step 1106, another constant γ′ may be used or another scale conversion function may be used for the calculation.

Expression (3) is a calculation expression in which the _section score score _section =1 when the following expression (4) is satisfied, and the section score score _section =3 when the following expression (5) is satisfied.

  FIG. 15 shows a graph for converting the average value of the scores of still images in the section obtained by Expression (3) into the level of the section. When the value of const in Expression (3) is changed, the average value of the scores of the still images in the section when the level of the section becomes 3, that is, the slope of the graph in FIG. 15 changes. In the present embodiment, the equation (3) is used such that the left half of the graph in the graph of FIG. 15 is a straight line with a constant inclination. The scope of the present invention is not limited to this, and the left half of the graph may be a curve defined by an exponential function or a higher-order function. Alternatively, a function such that the range of those straight lines or curves is biased to the left or right of the center of the horizontal axis, that is, the average of the scores of still images satisfying the formula (5) is smaller or larger. You may use the formula and constant which take a value.

The lane marking evaluation unit 115 divides the score of the section into predetermined stages to obtain the level of the section. In the present embodiment, the thresholds (1.67 and 2.33) are set so as to divide the range of the score of the _section (1≦score _section ≦3) into three equal parts, and when 1≦score _section ≦1.67. and level 3 when the level 2,2.33 <score _section ≦ 3 when the level 1,1.67 <score _section ≦ 2.33.

It should be noted that, while evaluating the section in which there are lane markings with a high degree of blurring that are difficult to visually recognize as level 3, and not to overestimate the section of level 2, the index γ of expression (2) and the const of expression (3) , And the above threshold may be adjusted. For example, in the present embodiment, of the six still images in the section, two still images detect a lane marking with a score of the still image score _frame = 3 ^γ, and the remaining four still images have lane markings. If it cannot be detected, the score of the section is score _section ≈ 2.34, and the above-mentioned various constants are set so that the section is evaluated as level 3. On the other hand, when level 2 lane markings are detected in all six still images, the score of the _section becomes score _section ≈1.83, and the above-mentioned various constants are set so that the section is appropriately evaluated as level 2. There is.

  The values of the indexes γ and const and the threshold of the score of the section are not limited to the above, and can be appropriately changed according to the number of still images to be grouped as a section.

  The score of the section may be used as it is as the evaluation (blindness) of the lane markings in the section without dividing the score of the section into stages of the level of the section.

  Next, the lane marking evaluation unit 115 repeats the processing from step 1101 for the next section as a processing target (step 1107).

  When there is no processing target section, the lane marking evaluation unit 115 outputs the blurring degree of the lane marking for each section obtained in step 1106, and ends the lane marking evaluation subroutine.

(Result output process)
FIG. 16 is a flowchart of the result output subroutine.

  When the result output subroutine is called, the output unit 116 determines whether the shape classification result is the road marking of the detection target (procedure 1601) for each partial image included in the partial image group (step 1601), and the gradation classification result is one of levels 1 to 3. It is determined whether there is any (step 1602). The output unit 116 sets a partial image whose shape classification result is a road marking to be detected and whose gray level classification results are levels 1 to 3 as a processing target of the subsequent procedure (step 1603). The partial image whose shape classification result is not the road marking of the detection target and the partial image of the road surface and the object which the gray level classification result does not include the road marking are not subjected to the processing in the following procedure (step 1604). As for the lane markings, the degree of blurring is obtained for each section, and therefore the partial image whose shape classification result is the lane markings is not the processing target of the following procedure.

  Thus, in addition to determining whether the partial image includes the road marking from the viewpoint of shape estimation, the device implicitly determines whether the partial image includes the road marking from the viewpoint of grayscale estimation. .. That is, it is determined whether the road marking is included from two different viewpoints. As a result, oversight of features due to the determination from only one viewpoint is reduced, and highly accurate determination is realized.

  Next, the output unit 116 outputs the identifier of the still image to which each partial image included in the partial image group belongs, the position and type in the still image, and the blurring degree (step 1605). The type is the class of the shape classification result, and the blurring degree is the class of the density classification result. Further, the position of the partial image in the still image may be any information as long as it can identify the area occupied by the partial image, for example, the coordinates of the upper left vertex of the partial image, the width and the height, or the upper left vertex of the partial image. Information on the coordinates and the coordinates of the lower right vertex may be used. Outputting the position in the still image may output the coordinates as numerical values, or may output the still image or the moving image in which a rectangular frame including the upper left vertex and the lower right vertex is drawn.

  The output unit 116 may evaluate the blurring degree of the road marking by outputting the level 1 to 3 of the grayscale classification result of the partial image by using the above-described formula (1) to evaluate the blurring degree of the road marking. ..

  Next, the output unit 116 outputs the blurring degree of the lane markings obtained by the lane marking evaluation subroutine for each section (step 1606). For example, the output unit 116 outputs the position information of the first and last still images included in the section, the type of lane markings, and the blurring degree of the lane markings. You may output the position in the still image in which the lane marking was detected.

  The output unit 116 ends the result output subroutine after outputting the information such as the blurring degree.

[Application example using output results]
The road marking defect detection apparatus 100 according to the present embodiment is provided with the function of determining and outputting the position and type of the road marking reflected in the still image in the still image and the degree of blurring. By improving the procedure shown above, not only the position in the still image of the road marking but also the shooting position is output, and the result is output to the map screen with a visualization device using, for example, Geographic Information System (GIS). Is also possible.

  For example, FIG. 17 is a diagram showing an example of a screen of an application using the output result of the road marking defect detection apparatus 100. The application screen 1700 shown in the figure includes a still image area 1701, a moving image area 1702, a map area 1703, a still image information area 1704, and a marking type selection area 1705.

  The still image area 1701 is an area for displaying a still image in which the road marking to be detected is detected. The positions of the partial images 1711 and 1712 in which the road markings have been detected are shown on the still image.

  The moving image area 1702 is an area in which the moving image used for detecting the road marking can be reproduced.

  The map area 1703 is an area for displaying a map of the place where the road marking is detected. A route 1730 at the time of shooting a moving image and markers 1731 to 1735 indicating the shooting position of the still image where the road marking is detected are displayed on the map. The markers 1731 to 1735 are displayed in a manner corresponding to the degree of blurring so that the degree of blurring of the detected road marking can be seen. For example, the blurring level is displayed in blue for level 1, yellow for level 2, and red for level 3. When any of the markers 1731 to 1735 is selected, the still image captured at the position of the selected marker 1731 to 1735 is displayed in the still image area 1701. In the example of FIG. 17, the marker 1732 is selected. The map displayed in the map area 1703 can be operated by enlarging/reducing, scrolling, etc., as in a general map application.

  In the still image information area 1704, the file name of the still image in which the road marking is detected is displayed. You may display only the file name of the still image in which the markers 1731 to 1735 are displayed on the map. When any one of the file names is selected, the still image with the selected file name is displayed in the still image area 1701. In the still image information area 1704, in addition to the file name, the type of the detected road marking, the degree of blurring, etc. may be displayed.

  The marking type selection area 1705 is used for narrowing down the information displayed in the map area 1703 and the still image information area 1704. Only the marker and file name of the still image in which the road marking of the type selected in the marking type selection area 1705 is detected are displayed in the map area 1703 and the still image information area 1704. For example, when only the pedestrian crossing notice check box is checked in the marking type selection area 1705, only the marker indicating the shooting position of the still image in which the pedestrian notice is detected and the file name are displayed. The blurring degree may be narrowed down in the marking type selection area 1705.

  FIG. 18 is a diagram showing an example of a screen of an application using the output result of the section of the road marking defect detection apparatus 100. The application screen 1800 shown in the figure includes a still image area 1801, a moving image area 1802, a map area 1803, a still image information area 1804, and a marking type selection area 1805.

  The still image area 1801 and the moving image area 1802 are the same as the still image area 1701 and the moving image area 1702 of FIG.

  In the map area 1803, sections 1831 to 1833 in which lane markings have been detected are displayed in an overlapping manner on the map. The sections 1831 to 1833 are displayed in a manner according to the degree of blurring so that the degree of blurring of the lane markings can be seen. When any of the sections 1831 to 1833 is selected, the still image captured at the position of the selected section 1831 to 1833 is displayed in the still image area 1801 and the moving image at the position corresponding to the selected section 1831 to 1833 is displayed. Is displayed in the moving image area 1802.

  In the still image information area 1804, the file names of the still images included in the selected sections 1831 to 1833 are displayed. When any one of the file names is selected, the still image with the selected file name is displayed in the still image area 1801.

  The marking type selection area 1805 is used for narrowing down the information to be displayed. Only the section in which the division line of the type selected in the marking type selection area 1805 is detected is displayed in the map area 1803.

  As described above, according to the present embodiment, the shape estimating unit 112 estimates the shape of the road marking from the photographed image of the road, classifies the presence or absence and the type of the road marking, and the shade estimating unit 113 , Estimating the shade of the road marking from the photographed image to classify the degree of blurring of the road marking, and the lane marking evaluation unit 115 evaluates the blurring degree of the lane marking for each section in which a plurality of photographed images are collected and outputs By outputting the degree of blurring of the marking line for each section, the unit 116 can determine the degree of blurring of the marking line for each section from the captured image of the road.

  The apparatus of the present invention can be realized by a computer and a program, and the program can be recorded in a recording medium or provided through a network.

100... Road marking defect detection device 111... Still image acquisition unit 112... Shape estimation unit 113... Shade estimation unit 114... Shooting position abutting unit 115... Marking line evaluation unit 116... Output unit 121, 123... Neural network 122, 124... Parameter 101, 102... Storage medium

Claims (13)

A road marking defect detection device for detecting a defect of the road marking from a photographed image of a road marking including at least one of road marking, marking line and non-legal display,
Shape estimating means for estimating the shape of the road marking from the photographed image and classifying the presence or absence and the type of the road marking,
A shade estimating means for estimating the shade of the color of the road marking from the photographed image classified as having the road marking and classifying the degree of blurring degree of the road marking,
When the road marking is a lane marking, a lane marking evaluating unit that evaluates the degree of blurring of the lane marking for each section in which the plurality of captured images are collected,
An output device that outputs the degree of blurring of the lane markings for each section. The road marking defect detection device according to claim 1,
The lane marking evaluation means calculates, for each of the plurality of photographed images in the section, a score of the photographed image based on the blurring degree of the lane markings in the photographed image, and totalizes the score based on the counting result. A road marking defect detecting device, characterized in that the degree of blurring of the lane markings is evaluated. The road marking defect detection device according to claim 2,
The road marking defect detection device, wherein the marking line evaluation means evaluates the photographed image with higher weighting as the degree of blurring of the marking line in the photographed image increases. The road marking defect detection device according to claim 3,
The higher the degree of blurring of the marking lines in the photographed image, the higher the weighted evaluation of the photographed image means that the score of the photographed image becomes exponentially higher as the degree of blurring of the marking lines in the photographed image increases. A pavement marking defect detection device for calculating the score of the photographed image. The road marking defect detection device according to any one of claims 2 to 4,
The marking line evaluating means compares the total value or the average value of the score of the photographed image in the section with a predetermined threshold value and evaluates the blurring degree of the marking line for each section. Detection device. The road marking defect detection device according to any one of claims 2 to 5,
The shape estimating unit acquires a portion of the captured image as a partial image, extracts a partial image of the partial image in which the road marking may exist as a processing target, and for each of the processing target partial images. Classify the presence and type of the road marking,
The gray level estimation means classifies the degree of blurring of the road marking for each of the partial images classified as having the road marking, and outputs a certainty factor for each step,
The lane marking evaluation means calculates the score of the partial image by weighting the certainty factor at the stage of the blurring degree with respect to the partial images classified by the grayscale estimation means, and calculates the score of the partial image, A road marking defect detecting device, characterized in that the score of the photographed image is calculated based on the score. The road marking failure detection device according to claim 6,
The road marking defect detection apparatus, wherein the lane marking evaluation means evaluates the partial images with higher weighting as the degree of blurring of the lane markings of the partial images increases, and calculates a score of the partial images. The road marking defect detection device according to claim 6 or 7, wherein
The shape estimating unit combines the plurality of partial images and re-forms them as a single partial image when it is estimated that a single road marking exists across a plurality of partial images. Road marking failure detection device. The road marking defect detection device according to any one of claims 6 to 8,
When the lane markings estimated to be the same are detected in a plurality of partial images in one captured image, the lane marking evaluation means calculates a score of the partial images for each of the partial images, and then the plurality of portions. A road marking defect detecting device, wherein the score of the photographed image is calculated based on a total value or an average value of the image scores. A road marking defect detection device for detecting a defect of the road marking from a photographed image of a road marking including at least one of road marking, marking line and non-legal display,
Acquiring a part of the captured image as a partial image, extracting a partial image of the partial image in which the road marking may exist as a processing target, the presence or absence of the road marking for each of the processing target partial images, and Shape estimation means for classifying the types,
While estimating the shade of color of the road marking from the partial image classified as having the road marking and classifying the stages of the degree of blurring of the road marking, a shade estimating means for outputting a certainty factor for each step,
The road marking is characterized in that: the gradation estimation means weights the degree of blurring with respect to the classified partial images with respect to the classified images, and outputs an output to evaluate and output the blurring degree of the road marking. Defect detection device. A road marking defect detection method executed by a road marking defect detecting device for detecting a defect of the road marking from a photographed image of a road marking,
A step of estimating the shape of the road marking including at least one of the road marking, the marking line, and the non-statutory display from the photographed image, and classifying the presence or absence and the type of the road marking,
A step of estimating the shade of color of the road marking from the photographed image classified as having the road marking and classifying the degree of blurring of the road marking;
When the road marking is a lane marking, a step of evaluating the blurring degree of the road marking for each section in which the plurality of captured images are collected,
And a step of outputting the degree of blurring of the road marking. A road marking defect detection method executed by a road marking defect detecting device for detecting a defect in the road marking from a photographed image of a road marking including at least one of road marking, marking line and non-legal display,
Acquiring a part of the captured image as a partial image, extracting a partial image of the partial image in which the road marking may exist as a processing target, the presence or absence of the road marking for each of the processing target partial images, and Classifying the types,
While estimating the shading degree of the road marking by estimating the shading of the color of the road marking from the partial image classified as having the road marking, outputting a certainty factor for each step,
A step of weighting the certainty factor at the stage of the degree of blurring with respect to the partial image to evaluate and output the degree of blurring of the road marking, and a method of detecting a road marking defect.   A road marking defect detection program, characterized in that a computer is operated as each means of the road marking defect detection device according to any one of claims 1 to 10.

Images