TWI573078B - Method and computer program product for recognizing road material and detecting holes - Google Patents

Description

Road material identification and pothole detection method and computer program product

The invention relates to a road material identification and a pothole detection method, and in particular to an identification method and a computer program product for combining a gene algorithm and an adaptive resonance theory network.

In recent years, with the rapid development of economy and technology, many countries have begun to use information and communication technology to enable more efficient use of transportation resources, such as navigation, traffic flow, traffic accident information, etc. These functions have increased traffic safety. , planning the shortest path, predicting traffic time and other benefits. In terms of traffic safety, the rugged road surface makes driving uncomfortable, and the holes on the road will endanger the safety of passers-by. In some conventional practices, image processing can be used to detect pits on the road surface, or because the temperature of the hole at night is higher than the surrounding temperature, temperature can also be used to judge the hole. A common practice is to add an acceleration sensor to the car and determine if there is a hole through the change in acceleration. How to improve these practices to accurately detect potholes on the road and provide better traffic information for Issues of interest to those skilled in the art.

Embodiments of the present invention provide a road material identification and pothole detection method suitable for an electronic device. The method comprises: obtaining a plurality of digital images of a plurality of roads, and extracting a plurality of feature vectors from the digital image; and inputting the feature vector into an adaptive resonance theoretical network, wherein the adaptive resonance theoretical network has a selection coefficient and a warning threshold Value and learning rate; the selection coefficient, the warning threshold and the learning rate are used as the chromosomes of the genetic algorithm, and the adaptive value of the genetic algorithm is determined according to the group correctness rate of the adaptive resonance theory network, thereby performing the genetic algorithm to determine Selection coefficient, warning threshold and learning rate; training adaptive resonance theory network according to selection coefficient, warning threshold, learning rate and eigenvector determined by genetic algorithm; according to the adaptive adaptive resonance theory network after training Identify the material of the test road; obtain multiple accelerations through the acceleration sensor, calculate the pothole information based on the acceleration; and transmit the material and pit information of the test road to the server.

In some embodiments, the step of determining the fitness value of the genetic algorithm based on the group correctness rate of the adaptive resonance theory network is performed according to the following equations (1), (2):

Where f _i is the fitness value of the i-th chromosome in the gene algorithm. NC _i is the number of clusters generated by training the adaptive resonance theoretical network based on the i-th chromosome. CR _i is the group correcting rate of the adaptive resonance theoretical network based on the i-th chromosome. The PC is a preset number of groups. C _j,k is the kth subgroup generated by training the adaptive resonance theoretical network according to the ith chromosome. D _k is the kth correct grouping. A is a set of feature vectors, and |A| is used to calculate the number of elements in set A.

In some embodiments, the adaptive resonance theoretical network includes a plurality of input layer nodes and a plurality of output layer nodes, the input layer nodes and the output layer nodes having a plurality of weights. The step of training the adaptive resonance theoretical network includes: expanding the feature vector, wherein the length of the extended feature vector is the same as the number of input layer nodes; according to the first extended feature vector, the selection coefficient and the expanded feature vector Weighting to select one of the output layer nodes; calculating the warning value according to the selected output layer node and the first extended feature vector, and determining whether the warning value is greater than the warning threshold; if the warning value is not a warning threshold, at the output layer A new node is added to the node; and if the alert value is greater than or equal to the alert threshold, the corresponding weight is updated according to the learning rate.

In some embodiments, the step of calculating the hole information according to the acceleration comprises: normalizing the acceleration according to an Euler angle; determining whether the acceleration on the Z axis is less than a preset value, and if so, setting the first algorithm to determine that a pit; determining whether the accelerations on the X-axis, the Y-axis, and the Z-axis are within a predetermined interval, and if so, setting a second algorithm to determine that there is a pit; and if the first algorithm is within a preset time range If there is a pothole determined by the second algorithm, the pothole information is set to have a pothole.

In some embodiments, the method further includes: obtaining the hole information from the server; and displaying the hole information in an augmented reality manner in the navigation service.

The embodiment of the present invention also provides a computer program product, which can perform the above-mentioned road material identification and pothole detection method after the computer program product is loaded into the electronic device for execution.

The above described features and advantages of the invention will be apparent from the following description.

110‧‧‧Digital imagery

120‧‧‧Electronic devices

121‧‧‧ processor

122‧‧‧ memory

123‧‧‧Acceleration sensor

200‧‧‧Adaptive Resonance Theory Network

201(1), 201(2), 201(3), 201(2M)‧‧‧ Input layer nodes

211(1), 211(2), 211(N)‧‧‧ output layer nodes

S301~S311, S401~S407‧‧‧ steps

FIG. 1 is a schematic diagram showing an electronic device according to an embodiment.

FIG. 2 is a schematic diagram showing an adaptive resonance theoretical network architecture according to an embodiment.

[Fig. 3A] and [Fig. 3B] are flowcharts showing a combined gene algorithm and an adaptive resonance theoretical network according to an embodiment.

FIG. 4 is a flow chart showing a road material identification and a pothole detection method according to an embodiment.

FIG. 1 is a schematic diagram of an electronic device according to an embodiment. Referring to FIG. 1 , the electronic device 120 includes a processor 121 , a memory 122 , and an acceleration sensor 123 . The electronic device 120 loads a computer program product into the memory 122, and the processor 121 executes the computer program product to execute the channel. Road material identification and pothole detection methods. The electronic device 120 receives a plurality of digital images 110 about the road. The digital images 110 may be training images for performing machine learning algorithms; the digital images 110 may also be test images, and the electronic device 120 may use the trained model. The material of the road in the digital image 110 is determined. In this embodiment, the material of the road can be divided into seven types: asphalt mixture, undried cement concrete, paving stone, crushed stone, industrial waste, soil of unfinished industrial roads, and potholes. On the other hand, the processor 121 also receives the acceleration detected by the acceleration sensor 123, thereby judging whether there is a pothole on the road. In some embodiments, the electronic device 120 is mounted on any form of vehicle, and the electronic device 120 can be implemented as a programmable circuit, a smart phone, or other suitable device, and the invention is not limited thereto. The road material identification and pothole detection methods will be described in detail below.

First, in the training phase, the electronic device 120 extracts a plurality of feature vectors from the digital image 110, such as a histogram of oriented gradients, a scale-invariant feature transform, or other The feature vector of luminance/chroma, the present invention does not limit the specific algorithm of these feature vectors. Before training, the feature vector must be normalized to the range of 0.05 to 0.95. This normalization can be written as the following equation (1).

Where x is a real number, representing an element in the feature vector, min represents the minimum value of the element, and max represents the maximum value of the element. The normalized eigenvectors can be expressed as I = ( a ₁ , a ₂ ..., a _M ), where a ₁ to a _M are both real numbers from 0.05 to 0.95, and M is a positive integer. Next, the feature vectors are extended, and the extended feature vector (also called the extended feature vector) can be expressed as ,among them . That is to say, the length of the extended feature vector is 2M.

Next, these extended feature vectors are input into the Adaptive Resonance Theory (ART) network. The adaptive resonance theory network is an unsupervised learning method. The advantage is that the dynamic competition learning network architecture can solve the problem of stability and plasticity conflict. Please refer to FIG. 2. FIG. 2 is a schematic diagram showing an adaptive resonance theoretical network architecture according to an embodiment. The adaptive resonance theory network 200 includes 2M input layer nodes 201(1), 201(2), 201(3)...201(2M), and N output layer nodes 211(1), 211(2) ... 211(N), where N is a positive integer. The connection between the input layer nodes 201(1)~201(2M) and the output layer nodes 211(1)~211(N) represents the weight, and the extended feature vector has the same length as the input layer node 201(1)~201. The number of (2M). Each of the input layer nodes 201(1)~201(2M) represents one element in the extended feature vector, and each of the output layer nodes 211(1)~211(N) represents a category, in this embodiment The material of the road. Next, an extended feature vector (also referred to as a first extended feature vector) may be randomly selected, and an output layer node 211(1) is selected according to the first extended feature vector, a selection coefficient, and a weight in the network. 211 (N), as shown in the following equation (2).

Where j=1~N, T _j represents a similar value corresponding to the jth output layer node 211(j). W _j represents a vector composed of 2M weights connected to the jth output layer node 211(j). α is called a selection coefficient and is a real number greater than zero. In particular, the operator is an AND logic of fuzzy logic, which can be defined as ( X ∧ Y ) _i ≡min( x _i , y _i ), meaning that vector X and vector Y are in fuzzy logic. The AND operation in is equivalent to calculating the minimum of the elements in the two vectors. On the other hand, the function Represents the sum of all elements in the computed vector X. In this embodiment, the output layer nodes having the largest similarity value T _j are selected, and the selected output layer nodes represent the categories to which the input feature vectors belong.

However, after finding the category to which the feature vector belongs, it must be judged whether the classification is appropriate. If the feature vector is not similar to the category to which it belongs, a category (ie, an output layer node) is added. Specifically, the warning value is calculated according to the selected output layer node and the extended feature vector, and it is determined whether the warning value is greater than a warning threshold, expressed as the following equation (3).

The left side of equation (3) is the warning value, and ρ is the warning threshold. If the alert value is not critical, a new node will be added to the existing output layer nodes 211(1)~211(N). It is worth noting that when the warning threshold is larger, the similarity of the feature vectors in the generated categories is higher, but the relative categories may also generate too many categories; if the warning threshold is too small, it may not be possible. A problem that presents differences between categories. Next, if the alert value is greater than or equal to the alert threshold, the corresponding weight W _j is updated according to a learning rate, which can be expressed as the following equation (4).

Where β is the learning rate, usually a real number between 0 and 1. Is the old weight of the jth output layer node 211(j), and For the new weight after the update. In general, if the learning rate is too large, the network structure may be unstable. If the learning rate is too small, the problem of poor network learning may occur. After updating the weights, you can enter the next feature vector until all feature vectors have been used.

As can be seen from the above description, the selection coefficient, the threshold value and the learning rate all affect the performance of the adaptive resonance theoretical network. In this embodiment, the three values are determined according to the genetic algorithm. Those skilled in the art will be able to understand gene algorithms, and only a few key points are described herein. Please refer to FIG. 3A and FIG. 3B . FIG. 3A and FIG. 3B are flowcharts illustrating a combined gene algorithm and an adaptive resonance theoretical network according to an embodiment. In step S301, an adaptation function is designed. Specifically, the adaptive function of the genetic algorithm can be determined according to the group correctness rate of the adaptive resonance theory network, and can be expressed as the following equations (5) and (6).

Where f _i is the fitness value of the i-th chromosome in the gene algorithm. NC _i is the number of clusters generated by training the adaptive resonance theoretical network based on the i-th chromosome. CR _i is the group correcting rate of the adaptive resonance theoretical network based on the i-th chromosome. The PC is a predetermined number of clusters, which is 7 in this embodiment. C _j,k is the kth subgroup generated by training the adaptive resonance theoretical network according to the ith chromosome. D _k is the kth correct grouping of the preset 7 subgroups. A is a set of all feature vectors, and |A| is used to calculate the number of elements in set A.

In step S302, the encoding mode is defined. Since the above selection coefficient, warning threshold and learning rate are both real numbers, a real type problem (rather than binary) is selected in this embodiment. In step S303, a chromosome is generated, and the selection coefficient, the warning threshold and the learning rate are combined into a vector, expressed as ( α , σ , β), and the vector can be used as the chromosome of the genetic algorithm. In step S304, an initial population is generated, for example, a plurality of chromosomes can be randomly generated. It is worth noting that each gene in each chromosome must satisfy the limit of the numerical range, in which K groups of chromosomes are generated, and K is a positive integer. . In step S305, the adaptive resonance theoretical network is trained based on the chromosomes. In step S306, the fitness value of each chromosome is calculated according to the above equations (5), (6).

In step S307, it is determined whether the termination condition is met. In this embodiment, it is determined whether the reproduction has exceeded a predetermined number of times, and if so, the termination condition is met. In step S308, a copy operation is performed, for example, the probability that the chromosome is picked is determined according to the following equation (7).

Where PS _i indicates the probability that the ith chromosome is selected. After the K chromosomes are copied, the mating of the chromosomes is performed (step S309 and step S310). In general, not all chromosomes are mated, and a mating rate is usually given to determine how many percentages of chromosomes to mate. If the mating rate is too high, excellent species may be lost; if the mating rate is too low, it is likely to cause stagnation during evolution, so the mating rate is usually set between 0.5 and 1. On the other hand, in general, there are three types of mating methods: simple mating, arithmetic mating, and heuristic mating. In order to avoid the possibility that the mated gene may exceed the numerical range, arithmetic mating is employed in step S309.

In step S310, a mutation operation is performed. Each gene on a chromosome has an equal chance of mutation, which is determined by the mutation rate. In order to avoid the loss of excellent chromosomes, the mutation rate is usually very small, generally between 0.01 and 0.1. In general, there are three types of mutations: uniform mutation, non-uniform mutation, and boundary mutation. Since the adaptive resonance theoretical network is sensitive to parameters close to the boundary value, boundary mutation is used in this embodiment.

In step S311, population substitution is performed. After the evolution of the case to produce the progeny chromosomes, the old chromosomes must be replaced by new ones to continue the evolution of the next generation. Generally, the group substitution method has a whole generation substitution and an elite strategy. In order to make the evolution process of the ethnic group retain the better genes of the old group and avoid completely losing the chromosome, the elite strategy is adopted in this embodiment.

Next, the adaptive resonance theoretical network is trained according to the selection coefficient determined by the genetic algorithm, the warning threshold, the learning rate, and the feature vectors, so that the training phase is ended, and each output layer is trained. The node represents the material of a road. In the test phase, the electronic device obtains the digital image of the test road, extracts the feature vector, and inputs the feature vector into the adaptive resonance theoretical network. After finding the output layer node with similar feature vectors, the electronic test device can know the test road. Material.

Referring to FIG. 1 , in addition to identifying the material of the road, the electronic device 120 obtains a plurality of accelerations through the acceleration sensor 123 , and calculates the hole information based on the accelerations. Specifically, the acceleration sensor 123 will obtain accelerations on the X-axis, the Y-axis, and the Z-axis. However, in some embodiments, the electronic device 120 is implemented as a smart phone, so that a rotation may occur during use. . In this embodiment, the electronic device 120 normalizes the accelerations according to the ruler angles. The Euler angles are mainly used to describe the rotation of the object in three dimensions, and the rotation of any object is from the coordinate system. The three Euler angles are rotated, so the vector of the object can be determined by three basic rotation matrices. In other words, any rotation matrix about the rotation of the object is composed of three basic rotation matrices. Taking the rotation around the X axis as an example, the electronic device 120 can be regarded as a reference when the X, Y, and Z axis angles are 0 degrees, and X', Y', and Z' are the vectors after the θ angle is rotated around the X axis. Vector transfer can be performed through Euler angles, as shown in the following equation (8).

In some embodiments, the electronic device 120 can obtain the angle θ of the electronic device 120 on the X-axis through some correction means, or can obtain the angle θ on the X-axis according to the preset setting angle of the electronic device 120, and then execute Equation (8). Similarly, the Euler angle can be normalized for both the Y and Z axes. In this way, the Euler angle can be used to normalize the acceleration when the electronic device 120 collects acceleration at an arbitrary angle.

Next, it can be judged based on the acceleration on the three axes whether or not there is a pit, and in this embodiment, the two algorithms are combined. In the first algorithm, it is determined whether the acceleration on the Z axis is less than a preset value, and if so, the first algorithm is set to determine that there is a pit. Since the vehicle first falls and then rises when the pit is encountered, in some embodiments, the Z-axis acceleration when collecting the pits is first collected, and the preset value is determined according to the following equation (9).

Where T is the preset value. N is the number of driving times. g _z,i,j is the acceleration of the jth Z-axis of the i-th driving. e _i is the record of entering the pothole in the ith driving, l _j is the record of leaving the pothole in the ith driving. The first algorithm can determine the situation when the vehicle falls into the pothole.

In the second algorithm, it is determined whether the accelerations on the X-axis, the Y-axis, and the Z-axis are all within a predetermined interval, and if so, the second algorithm is set to determine that there is a hole. The lower limit of this preset interval can be calculated according to the following equation (10), and the upper limit can be calculated according to the equation (11).

If the first algorithm and the second algorithm determine that there is a pothole in the preset time range, it is determined that there is a pothole, and the electronic device 120 generates a pothole information and marks the position of the pothole therein. Finally, the electronic device 120 can transmit the material of the test road and the hole information to a server (not shown). In some embodiments, the electronic device 120 also has a display screen (not shown) to provide navigation. The electronic device 120 can obtain the hole information from the server and display the pit in an augmented reality manner in the navigation service. Hole information, so users can instantly know where there are holes. However, the present invention does not limit the way in which the interface of the augmentation is augmented and the information of the potholes is displayed.

4 is a flow chart showing a road material identification and a pothole detection method according to an embodiment. In step S401, a digital image about the road is acquired, and a feature vector is extracted from the digital image. In step S402, the feature vector is input into an adaptive resonance theoretical network having a selection coefficient and a warning gate. Depreciation and learning rate. In step S403, the selection coefficient, the warning threshold and the learning rate are used as the chromosomes of the genetic algorithm, and the adaptive value of the genetic algorithm is determined according to the group correcting rate of the adaptive resonance theory network, thereby performing the genetic algorithm to determine Selection factor, warning threshold and learning rate. In step S404, the adaptive resonance theoretical network is trained according to the selection coefficient, the alert threshold, the learning rate, and the feature vector determined by the genetic algorithm. In step S405, the material of the test road is identified based on the trained adaptive resonance theoretical network. In step S406, a plurality of accelerations are acquired by the acceleration sensor, and the hole information is calculated based on the acceleration. In step S407, the material of the test road and the pothole information are transmitted to the server. The steps in Fig. 4 have been described in detail above and will not be described again here. The method of Figure 4 can be used in conjunction with the above embodiments, or it can be used alone. In other words, other steps can be added between the steps of FIG.

The invention also proposes a computer program product, which can execute the above road material identification and pothole detection method after the computer program product is loaded into the electronic device and executed.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S401~S407‧‧‧Steps

Claims (8)

A road material identification and pothole detection method is applicable to an electronic device, the method comprising: acquiring a plurality of digital images of a plurality of roads, and extracting a plurality of feature vectors from the digital images; and the feature vectors Inputting an adaptive resonance theoretical network, wherein the adaptive resonance theoretical network has a selection coefficient, a warning threshold and a learning rate; the selection coefficient, the warning threshold and the learning rate are used as a genetic algorithm a chromosome, and determining an adaptation value of the genetic algorithm according to a group correctness rate of the adaptive resonance theory network, thereby performing the gene algorithm to determine the selection coefficient, the warning threshold and the learning rate; The selection coefficient, the alert threshold, the learning rate, and the feature vectors determined by the genetic algorithm train the adaptive resonance theoretical network; and identify a test according to the trained adaptive resonance theoretical network The material of the road; obtaining a plurality of accelerations through an acceleration sensor, and calculating a hole information based on the accelerations; The material of the test road and the hole information are transmitted to a server, wherein the step of calculating the hole information according to the accelerations comprises: normalizing the accelerations according to an Euler angle; determining the Whether the acceleration is less than a preset a value, if yes, setting a first algorithm to determine that there is a hole; determining whether the accelerations on the X axis, the Y axis, and the Z axis are within a predetermined interval, and if so, setting a second algorithm to determine a pothole; and if the first algorithm and the second algorithm determine that there is a pothole within a predetermined time range, the pit information is set to have a pothole. The road material identification and pothole detection method according to claim 1, wherein the step of determining the fitness value of the genetic algorithm according to the group correctness rate of the adaptive resonance theory network is according to the following equation ( 1), (2) executed: Where f _i is the fitness value of the i-th chromosome in the algorithm of the gene, and NC _i is the number of clusters generated by training the adaptive resonance theoretical network according to the i-th chromosome, and CR _i is based on the i-th The chromosome trains the correct correct rate of the cluster of the adaptive resonance theoretical network, PC is a predetermined number of clusters, and C _j,k is the kth cluster generated by training the adaptive resonance theoretical network according to the ith chromosome , D _k is the k th correct grouping, a set of feature vectors for some, | a | a set is used for calculating the number of elements. The road material identification and pothole detection method according to claim 2, wherein the adaptive resonance theoretical network comprises a plurality of input layer nodes and a plurality of output layer nodes, and the input layer nodes and the outputs Having a plurality of weights between the layer nodes, wherein the step of training the adaptive resonance theory network comprises: expanding the feature vectors, wherein the lengths of the extended feature vectors are the same as the number of the input layer nodes; Extracting one of the output layer nodes by using a first extended feature vector, the selection coefficient and the weights of the expanded feature vectors; and selecting the output layer node and the first extended feature vector according to the selected Calculating an alert value and determining whether the alert value is greater than the alert threshold; if the alert value is not greater than the alert threshold, adding a new node to the output layer nodes; and if the alert value is greater than or equal to the alert The threshold value is updated according to the learning rate. The road material identification and pothole detection method described in claim 1 further includes: obtaining the hole information from the server; and displaying the hole information in an augmented reality manner in a navigation service . A computer program product for loading into an electronic device The electronic device executes the computer program product to perform a plurality of steps: obtaining a plurality of digital images of the plurality of roads, and extracting a plurality of feature vectors from the digital images; and inputting the feature vectors into an adaptive resonance a theoretical network, wherein the adaptive resonance theory network has a selection coefficient, an alert threshold, and a learning rate; the selection coefficient, the alert threshold, and the learning rate are used as a chromosome of a genetic algorithm, and A group correctness rate of the adaptive resonance theory network determines an adaptation value of the gene algorithm, thereby executing the gene algorithm to determine the selection coefficient, the warning threshold value and the learning rate; according to the gene algorithm Determining the selection coefficient, the warning threshold, the learning rate, and the feature vectors to train the adaptive resonance theoretical network; identifying the material of a test road according to the trained adaptive resonance theoretical network; An acceleration sensor obtains a plurality of accelerations, and calculates a hole information based on the accelerations; and the test road And the hole information is transmitted to a server, wherein the step of calculating the hole information according to the accelerations comprises: normalizing the accelerations according to an Euler angle; determining whether the accelerations on the Z axis are smaller than a preset value, if yes, setting a first algorithm to determine that there is a hole; determining whether the accelerations on the X axis, the Y axis, and the Z axis are within a predetermined interval, and if so, setting a second calculation Judgment There is a pothole; and if the first algorithm and the second algorithm determine that there is a pothole within a predetermined time range, the pothole information is set to have a pothole. The computer program product of claim 5, wherein the step of determining the fitness value of the genetic algorithm according to the group correctness rate of the adaptive resonance theory network is according to the following equations (1), (2) Executed: Where f _i is the fitness value of the i-th chromosome in the algorithm of the gene, and NC _i is the number of clusters generated by training the adaptive resonance theoretical network according to the i-th chromosome, and CR _i is based on the i-th The chromosome trains the correct correct rate of the cluster of the adaptive resonance theoretical network, PC is a predetermined number of clusters, and C _j,k is the kth cluster generated by training the adaptive resonance theoretical network according to the ith chromosome , D _k is the k th correct grouping, a set of feature vectors for some, | a | a set is used for calculating the number of elements. The computer program product of claim 6, wherein the adaptive resonance theory network comprises a plurality of input layer nodes and a plurality of output layer nodes, and the input layer nodes and the output layer nodes are Having a plurality of weights, wherein the step of training the adaptive resonance theoretical network comprises: expanding the feature vectors, wherein the expanded feature vectors have the same length as the number of the input layer nodes; a first extended feature vector of the feature vectors, the selection coefficient and the weights to select one of the output layer nodes; and calculating the output node node and the first extended feature vector according to the selected one The warning value is determined whether the warning value is greater than the warning threshold; if the warning value is not greater than the warning threshold, a new node is added to the output layer nodes; and if the warning value is greater than or equal to the warning threshold, according to The learning rate updates the corresponding weights. The computer program product of claim 5, wherein the steps further comprise: obtaining the hole information from the server; and displaying the hole information in an augmented reality manner in a navigation service.