CN106709412A - Traffic sign detection method and apparatus - Google Patents
Traffic sign detection method and apparatus Download PDFInfo
- Publication number
- CN106709412A CN106709412A CN201510797278.0A CN201510797278A CN106709412A CN 106709412 A CN106709412 A CN 106709412A CN 201510797278 A CN201510797278 A CN 201510797278A CN 106709412 A CN106709412 A CN 106709412A
- Authority
- CN
- China
- Prior art keywords
- traffic sign
- candidate region
- significant
- view image
- street view
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention relates to a traffic sign detection method and apparatus. The method comprises the following steps: obtaining a street scenery image; obtaining candidate areas matching a color value scope of a sign figure in a traffic sign from the street scenery image; screening the obtained candidate areas, wherein screened candidate areas accord with preset area features of an area where the sign figure is disposed; and after features of the screened candidate areas are extracted, performing discrimination on the extracted features through a classifier used for determining whether the features belong to a sign figure type so as to obtain a traffic sign detection result. The traffic sign detection method and apparatus provided by the invention can realize automatic traffic sign detection, require no manual obtaining and improve the efficiency and accuracy.
Description
Technical field
The present invention relates to technical field of image detection, more particularly to a kind of method for traffic sign detection and device.
Background technology
Traffic sign refers to the object with traffic alarm or traffic suggesting effect set up by street, such as area
Between test the speed mark, traffic prohibited sign, road condition advisory mark or current direction sign etc..By
The position of traffic sign is marked in electronic map, the retrieval and navigation to road information are all significant.
However, the acquisition of information source of traffic sign mainly needs by artificial acquisition, but artificial acquisition at present
Want a large amount of manpowers and consume the plenty of time, efficiency is too low.And it is poor manually to obtain accuracy.
The content of the invention
Based on this, it is necessary to which the acquisition of information source for current traffic sign relies primarily on artificial acquisition and causes
The problem that efficiency is low and accuracy is poor, there is provided a kind of method for traffic sign detection and device.
A kind of method for traffic sign detection, methods described includes:
Obtain street view image;
Obtain what is matched with the color span of the significant figure in traffic sign from the street view image
Candidate region;
Candidate region to obtaining is screened, and the candidate region for filtering out meets where the significant figure
The predeterminable area feature in region;
After feature is extracted in the candidate region that will be filtered out, by the feature of extraction by for discriminating whether to belong to mark
The grader of will graphics class is differentiated, to obtain road traffic sign detection result.
A kind of road traffic sign detection device, described device includes:
Street view image acquisition module, for obtaining street view image;
Candidate region acquisition module, for being obtained from the street view image and the significant figure in traffic sign
The candidate region of the color span matching of shape;
Candidate region screening module, for being screened to the candidate region for obtaining, the candidate region for filtering out
Meet the predeterminable area feature of the significant figure region;
Discrimination module, after feature is extracted in the candidate region for that will filter out, by the feature of extraction by being used for
Discriminate whether that the grader for belonging to significant graphics class is differentiated, to obtain road traffic sign detection result.
Above-mentioned method for traffic sign detection and device, after street view image is got, according to significant figure
Color span obtains candidate region, can so be filtered out from street view image and substantially not meet traffic mark
The region of the color characteristic of will.The preset areas for meeting significant figure region are filtered out from candidate region
The candidate region of characteristic of field, so can further filter out the predeterminable area spy for substantially not meeting traffic sign
The candidate region levied.By the differentiation of grader, may finally detect in street view image with the presence or absence of mark
Property figure, so as to detect whether there is corresponding traffic sign.The traffic mark of automation can so be realized
Will detects that, without artificial acquisition, efficiency and accuracy all get a promotion.
Brief description of the drawings
Fig. 1 be one embodiment in for realize method for traffic sign detection computer cut-away view;
Fig. 2 is the schematic flow sheet of method for traffic sign detection in one embodiment;
Fig. 3 is the schematic diagram of the interval mark that tests the speed in one embodiment;
Fig. 4 is the schematic diagram of the interval mark that tests the speed in another embodiment;
Fig. 5 is the schematic flow sheet of method for traffic sign detection in another embodiment;
Fig. 6 is the schematic diagram of horizontal 360-degree panorama street view image in one embodiment;
Fig. 7 is that test the speed the outline drawing of mark in hsv color model interval in one embodiment;
Fig. 8 is in binary image in one embodiment after connecting domain lookup at the interval mark that tests the speed
The schematic diagram of image;
Fig. 9 is the schematic diagram of the significant figure of traffic sign detected in one embodiment;
Figure 10 is the structured flowchart of road traffic sign detection device in one embodiment;
Figure 11 is the structured flowchart of road traffic sign detection device in another embodiment.
Specific embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, below in conjunction with accompanying drawing and reality
Example is applied, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only
Only it is used to explain the present invention, is not intended to limit the present invention.
As shown in figure 1, in one embodiment, there is provided a kind of computer 100, including by system bus
The processor of connection, non-volatile memory medium, built-in storage and image acquisition device.Wherein process utensil
There is the function of computing function and the work of control computer 100.The processor is configured as performing a kind of traffic mark
Will detection method.Non-volatile memory medium includes that the storage of magnetic storage medium, optical storage media and flash memory type is situated between
At least one in matter, non-volatile memory medium is stored with operating system and a kind of road traffic sign detection device,
The road traffic sign detection device is used to realize a kind of method for traffic sign detection.Image acquisition device is used to gather reality
When image, image acquisition device can be 360 degree of full-view cameras.
As shown in Fig. 2 in one embodiment, there is provided a kind of method for traffic sign detection, the present embodiment
The computer 100 that is applied in this way in above-mentioned Fig. 1 is illustrated.The method specifically includes following step
Suddenly:
Step 202, obtains street view image.
Specifically, computer 100 can call image acquisition device to obtain street view image, the street view image of acquisition
Can be described as original street view image.Image acquisition device may be provided on movable fixture, so can be by can
Mobile device obtains street view image in real time in moving process.Movable fixture include automobile, unmanned plane and
At least one in robot.Street view image refer to road by image image.Road includes highway
With Ren Hang streets, highway include highway.
In one embodiment, street view image is horizontal 360-degree panorama street view image.Wherein horizontal 360-degree
Panorama street view image is to include covering 360 degree of street view images in the visual field in direction of horizontal plane.Computer 100 can
The street view image of control image acquisition device collection multiple directions synthesizes 360 street view images, it is also possible to control figure
As simultaneously collector collection horizontal rotation simultaneously obtains street view image and synthesizes to obtain 360 street view images.
Step 204, obtains the color span with the significant figure in traffic sign from street view image
The candidate region matched somebody with somebody.
The step of step 204 is color filtering.Wherein, traffic sign refers to be set up by street with traffic
Warning or the culture of traffic suggesting effect, such as test the speed mark, traffic prohibited sign, road conditions of interval are carried
Indicating will or current direction sign etc..Test the speed 3a, 3b and the 3c indicated as in Fig. 3 in specific interval.
Significant figure refers to the figure for distinguishing the traffic sign and other traffic signs in traffic sign
Shape.For example, when traffic sign tests the speed mark for interval, significant figure tests the speed in mark for interval
Including camera image and do not include the image of word, the 3a1 in such as Fig. 3.
The shape of traffic sign mainly includes rectangle, circular and triangle, therefore traffic sign shape is deposited in itself
In certain general character, distinction is not fine, realizes handing over here using the significant figure in traffic sign
The detection of logical mark, accuracy rate is higher.
Traffic sign is different from spontaneous object, and the color composition of specific traffic sign is fixed.
Specific interval is tested the speed, and mark is main to be made up of green and white, also some intervals test the speed mark it is main by blue and
White composition, and some intervals are tested the speed, mark is mainly made up of yellow and black.Interval survey as shown in Figure 4
Speed mark includes significant figure 400, and the region 401 and 402 of significant figure 400 is green, region
403 is white.
Change in view of light in natural environment causes the color that significant figure is presented also to change,
Here the color that significant figure is presented under various light conditions can be in advance counted, so as to true according to statistics
The fixed color span.Color span can be drawn according to the Color Channel under specific color model
Point, if than street view image use hsv color model, then color span can include H (Hue,
Tone), in three Color Channels of S (Saturation, saturation degree) and V (Value, brightness) at least one
The span planted.Color model can also use RGB (RGB) color model or YUV colors
Model etc..
Step 206, the candidate region to obtaining is screened, and the candidate region for filtering out meets described significant
The predeterminable area feature of figure region.
Specifically, the step of color filtering after acquired candidate region still can include more and traffic
Indicate incoherent candidate region, filtered here by the candidate region for obtaining, filter out and do not meet
The candidate region of predeterminable area feature, so as to obtain the candidate region for meeting predeterminable area feature.Predeterminable area
It is characterized in default Regional Characteristics that significant figure region should have.
Step 208, after feature is extracted in the candidate region that will be filtered out, be by for differentiation by the feature of extraction
The no grader for belonging to significant graphics class is differentiated, to obtain road traffic sign detection result.
Specifically, grader is by belonging to the positive sample collection of significant graphics class and being not belonging to significant graphics class
Negative sample collection training.The positive sample that wherein positive sample is concentrated is the image for including significant figure;Negative sample
The negative sample of concentration is then not include the image of significant figure, and negative sample can be included with significant figure not
Related random image, it is also possible to including the image similar to significant figure.
During training grader, the negative sample that the positive sample and negative sample that positive sample is concentrated are concentrated extracts spy respectively
After levying, grader is trained according to the feature extracted.Trained grader can be used to predict one it is new
Whether image belongs to significant graphics class, illustrates to detect traffic sign if significant graphics class is belonged to;
Illustrate to be not detected by traffic sign if significant graphics class is not belonging to.Road traffic sign detection result includes
It is no to detect traffic sign, position of the traffic sign for detecting in street view image can also be included.
The feature wherein extracted can use HOG (Histogram of Oriented Gradient, direction gradient
Histogram) feature, SIFT (Scale-Invariant Feature Transform, scale invariant feature conversion) be special
Levy or SURF (Speeded Up Robost Features, accelerate robust feature) feature etc..Grader can
With using SVM (Support Vector Machine, SVMs), cascade classifier (such as Adaboost
Grader) or artificial nerve network classifier etc..
Above-mentioned method for traffic sign detection, after street view image is got, takes according to the color of significant figure
Value scope obtains candidate region, and the face for substantially not meeting traffic sign can be so filtered out from street view image
The region of color characteristic.The predeterminable area feature for meeting significant figure region is filtered out from candidate region
Candidate region, so can further filter out substantially do not meet traffic sign predeterminable area feature time
Favored area.By the differentiation of grader, may finally detect with the presence or absence of significant figure in street view image,
So as to detect whether there is corresponding traffic sign.The road traffic sign detection of automation can be so realized,
Without manually obtaining, efficiency and accuracy all get a promotion.
In one embodiment, computer 100 can also obtain the geographical location information of street view image, and root
The position in street view image and the geographical location information according to the traffic sign for detecting, on the electronic map
Mark corresponding traffic sign.Wherein the geographical location information of street view image can be obtained when street view image is obtained.
In one embodiment, computer 100 can also carry out prompting early warning according to road traffic sign detection result.
Such as send the prompting of " testing the speed in front " after mark detecting interval and test the speed.
In one embodiment, before step 204, also include:According to traffic sign in pre-acquiring streetscape figure
The statistical information of the appearance position as in, cuts to street view image.
Specifically, pre-acquiring street view image refers to the advance street view image for obtaining, by pre-acquiring streetscape figure
The appearance position of traffic sign carries out statistics and obtains corresponding statistical information as in.The statistical information can reflect
Go out traffic sign and do not appear in which position of street view image, or be likely to appear in street view image which
A little positions.
Such as, usual street view image can include the image of sky and the image of road, the usual position of image of sky
In the upper area of street view image, the image of road is then located at the lower area of street view image, in these regions
Be not in traffic sign, these regions can be cropped.
In the present embodiment, according to traffic sign in pre-acquiring street view image appearance position statistical information come
Street view image is cut, can directly be abandoned in flow front end the region of traffic sign can not possibly occurs,
So as to improve the efficiency of subsequent treatment.
In one embodiment, step 204 includes:By in street view image with traffic sign in significant figure
The pixel value of the color span matching of shape is set to prospect colour, will be unmatched with the color span
Pixel value is set to background colour, obtains binary image;Prospect colour is searched in the binary image to constitute
Connected domain getting candidate region.
Connection domain lookup wherein is carried out to the binary image, so as to obtain what is be made up of the pixel of prospect colour
Connected domain, using the connected domain as candidate region, or by the connected domain corresponds to original background image
Pixel as candidate region.
Further, prospect colour can be 0, be visually appear as black;Background colour can be 255, vision
On show as white.Candidate region matches with color span, refers to the candidate region in street view image
Corresponding pixel value is in the color span.
Connected domain refers to the image district with the adjacent foreground pixel point composition of same pixel value and position in image
Domain.Connection domain lookup is the process that the connected domain in image is found out and marked, also referred to as connected domain analysis.
Connection domain lookup can use Two-Pass (two-pass scan) algorithms and Seed Filling (seed filling) algorithm.
The angle of connected domain can also be adjusted after connected domain is found so that angle is standardized.Such as will
Angle adjustment is to [- 90 °, 90 °] in the range of this.
In one embodiment, step 204 includes:Obtained from street view image and the mark in traffic sign
Property figure a kind of composition color pre-set color span matching candidate region.The one kind constitutes color
Can use the color of area accounting maximum in significant figure.Such as traffic sign for interval test the speed mark when, face
Color span can be that the color of camera image is green.By a kind of color value model for constituting color
Enclose to obtain candidate region, efficiency can be improved and retain marginal information.
In one embodiment, step 204 includes:Obtained from street view image and the mark in traffic sign
Property figure a kind of composition color multigroup pre-set color span at least one set of matching candidate regions
Domain.
Specifically, it is contemplated that outdoor optical is larger according to condition difference, the change of color span is also larger, here
In advance for a kind of composition color sets multigroup pre-set color span, as long as certain region hit one of which is pre-
If color span can be just retained as candidate region, if being unsatisfactory for any group of pre-set color value
Scope just filters out the region.
For example, if traffic sign is to test the speed mark in interval, a kind of composition color of significant figure is green
Color, then a kind of multigroup pre-set color span of composition color of the significant figure in traffic sign is specific
Can be as listed below:
First group:50<H<100, S>55,75<V<160.
Second group:50<H<100, S>50, V<70.
3rd group:60<H<90, S>35,170<V<220.
4th group:28<H<100,110<V<178.
5th group:20<H<110,130<S<230,30<V<60.
Wherein H, S and V represent tonal color passage, saturation degree Color Channel and lightness colors passage respectively
Under pixel value.
In the present embodiment, by setting a kind of multigroup pre-set color span for constituting color, it can be ensured that
Detection of the color filtering step to traffic sign has recall rate higher, prevents missing inspection.Wherein recall rate is again
Claim recall ratio, be the traffic sign for detecting quantity account for physical presence traffic sign quantity ratio.
In one embodiment, predeterminable area feature includes:The size range of significant figure region,
At least one in depth-width ratio scope, color cluster classification number and Gradient Features.
The size range of wherein significant figure region refers to the circumscribed rectangle of significant figure region
Width scope and the scope of height.If than traffic sign to test the speed mark in interval, and the streetscape for obtaining
Picture size is 8192 pixels (width) × 4096 pixel (height), then the interval is tested the speed and indicates the chi of region
Very little scope can be:14 pixels<height<231 pixels, 10 pixels<width<600 pixels;Wherein height
Represent that the interval is tested the speed and indicate the height of region, width represents that the interval is tested the speed and indicates the width of region
Degree.
Depth-width ratio scope refers to the model of the ratio of the height and width of the circumscribed rectangle of significant figure region
Enclose.If than traffic sign to test the speed mark in interval, then test the speed the depth-width ratio model of mark region in the interval
Enclosing to be:1.0<height/width<5.2;Wherein height represents that the interval is tested the speed and indicates the height of region
Degree, width represents that the interval is tested the speed and indicates the width of region.
Color cluster classification number refers to be obtained after the pixel value of significant figure region is clustered
Categorical measure.Cluster can use K-means clustering algorithms, can also be using K-modes and other clusters
Algorithm, will not enumerate.
If than traffic sign to test the speed mark in interval, then K-means clustering algorithms can be used, first from obtaining
Arbitrarily K pixel of selection is used as initial cluster center in the candidate region for taking;And for remaining other pixels,
Then according to their pixel value distances with these cluster centres, closest cluster is assigned these to respectively
Cluster representated by center;Then each cluster centre for obtaining new cluster is calculated again;Constantly repeat this mistake
Journey is untill canonical measure function starts convergence.If it is two classes that can finally cluster, retain corresponding time
Favored area;If it is two classes that can not finally cluster, corresponding candidate region is filtered out.Wherein pixel value
Distance can be calculated using Euclidean distance.Canonical measure function can use mean square deviation.
If image is regarded as two-dimensional discrete function, the gradient of the image is the derivation of this two-dimensional discrete function,
Gradient can reflect the edge of image, and Gradient Features can reflect edge feature in image.If than handing over
The logical interval that is masked as is tested the speed mark, because traffic sign includes a large amount of green areas, is easily mixed with green tree leaf
Confuse.But edge is regular, negligible amounts in the significant figure of traffic sign, and the edge of leaf is then
It is mixed and disorderly, a fairly large number of, so will can be just indicated by the way that the Gradient Features of conversion change can be reflected
Property figure and leaf are distinguished.
In above-described embodiment, can be from the multiple of area size, depth-width ratio, colour type and Gradient Features
Dimension is filtered to the candidate region for obtaining, and filters out the candidate region for substantially not meeting predeterminable area feature,
Reservation there may be the candidate region of significant figure, be easy to improve the effect differentiated subsequently through grader
Rate.And the combination of the predeterminable area feature for passing through multiple dimensions, can cause to be carried out subsequently through grader
The efficiency of differentiation is further lifted.
As shown in figure 5, in one embodiment, there is provided a kind of method for traffic sign detection, the present embodiment
Middle traffic sign is to test the speed mark in interval.The method specifically includes following steps:
Step 502, obtains horizontal 360-degree panorama street view image.Such as horizontal 360-degree panorama street view image can
As shown in fig. 6, the high order end of the horizontal 360-degree panorama street view image can be seamlessly connected with low order end.
Step 504, the appearance position according to traffic sign in pre-acquiring horizontal 360-degree panorama street view image
Statistical information, the horizontal 360-degree panorama street view image to obtaining cuts.Reference picture 6, horizontal 360-degree
The region 601 and region 602 of panorama street view image can crop.
Step 506, under hsv color model, 360 degree of each Color Channels of panorama street view image of detection level
In each pixel value whether with traffic sign in significant figure green multigroup pre-set color span in
At least one set of matching.Wherein under hsv color model, mark is tested the speed as shown in Figure 7 in interval.Test the speed in interval
Multigroup pre-set color span of the significant figure of mark can use five groups listed below:
First group:50<H<100, S>55,75<V<160.
Second group:50<H<100, S>50, V<70.
3rd group:60<H<90, S>35,170<V<220.
4th group:28<H<100,110<V<178.
5th group:20<H<110,130<S<230,30<V<60.
Step 508, the pixel value that matching will be detected as in horizontal 360-degree panorama street view image is set to prospect colour,
Unmatched pixel value will be detected as in horizontal 360-degree panorama street view image and be set to background colour, obtain two-value
Change image.
Step 510, connection domain lookup is carried out to binary image and obtains candidate region.Specifically looked into by connected domain
Image in binary image after looking at the interval mark that tests the speed is as shown in Figure 8.Wherein above-mentioned steps 506 to
Step 510 be under hsv color model, from the street view image obtain with traffic sign in it is significant
The step of candidate region of at least one set of matching in green multigroup pre-set color span of figure.
Step 512, filters out the size model for not meeting significant figure region from the candidate region for obtaining
Enclose, the candidate region of any one in depth-width ratio scope, color cluster classification number and Gradient Features.
Wherein size range can use 14 pixels<height<231 pixels, 10 pixels<width<600 pixels.It is high
It is wide more desirable by 1.0 than scope<height/width<5.2.Wherein height represents that the interval is tested the speed and indicates region
Height, width represent the interval test the speed mark region width.Color cluster classification number can use 2 classes.
Here Gradient Features are mainly the Gradient Features distinguished with the Gradient Features of leaf.
Step 514, after HOG features are extracted in the candidate region that will retain after filtering, the HOG that will be extracted is special
Levy and differentiated by for the SVMs for discriminating whether to belong to significant graphics class, to obtain traffic mark
Will testing result.The significant figure of the final traffic sign for detecting 9a and 9b as shown in Figure 9.
As shown in Figure 10, in one embodiment, there is provided a kind of road traffic sign detection device 1000, have
Realize the functional module of the method for traffic sign detection of above-mentioned each embodiment.The road traffic sign detection device
1000 include:Street view image acquisition module 1001, candidate region acquisition module 1002, candidate region screening mould
Block 1003 and discrimination module 1004.
Street view image acquisition module 1001, for obtaining street view image.
Specifically, street view image acquisition module 1001 can be used to call image acquisition device to obtain street view image,
The street view image of acquisition can be described as original street view image.Image acquisition device may be provided on movable fixture,
So street view image can in real time be obtained in moving process by movable fixture.Movable fixture includes vapour
At least one in car, unmanned plane and robot.Street view image refer to road by image image.
Road includes highway and Ren Hang streets, and highway includes highway.
In one embodiment, street view image is horizontal 360-degree panorama street view image.Wherein horizontal 360-degree
Panorama street view image is to include covering 360 degree of street view images in the visual field in direction of horizontal plane.Street view image is obtained
Module 1001 can be used to control the street view image of image acquisition device collection multiple directions to synthesize 360 streetscape figures
Picture, it is also possible to control image acquisition device to gather horizontal rotation and obtain street view image simultaneously and synthesize to obtain 360
Street view image.
Candidate region acquisition module 1002, for being obtained from street view image and the significant figure in traffic sign
The candidate region of the color span matching of shape.
Wherein, traffic sign refers to the artificiality with traffic alarm or traffic suggesting effect set up by street
Body, such as interval test the speed mark, traffic prohibited sign, road condition advisory mark or current direction sign
Deng.Significant figure refers to the figure for distinguishing the traffic sign and other traffic signs in traffic sign.
For example, when traffic sign for interval test the speed mark when, significant figure can for interval test the speed mark in wrap
Include camera image and the not image including word, the 3a1 in such as Fig. 3.
The shape of traffic sign mainly includes rectangle, circular and triangle, therefore traffic sign shape is deposited in itself
In certain general character, distinction is not fine, realizes handing over here using the significant figure in traffic sign
The detection of logical mark, accuracy rate is higher.
Traffic sign is different from spontaneous object, and the color composition of specific traffic sign is fixed.
Specific interval is tested the speed, and mark is main to be made up of green and white, also some intervals test the speed mark it is main by blue and
White composition.Test the speed mark in interval as shown in Figure 4, and region 401 is green, and region 402 is white.
Change in view of light in natural environment causes the color that significant figure is presented also to change,
Here the color that significant figure is presented under various light conditions can be in advance counted, so as to true according to statistics
The fixed color span.Color span can be drawn according to the Color Channel under specific color model
Point, if than street view image use hsv color model, then color span can include H (Hue,
Tone), in three Color Channels of S (Saturation, saturation degree) and V (Value, brightness) at least one
The span planted.Color model can also use RGB (RGB) color model or YUV colors
Model etc..
Candidate region screening module 1003, for being screened to the candidate region for obtaining, the candidate for filtering out
Region meets the predeterminable area feature of the significant figure region.
Specifically, still more and traffic sign can be included not by candidate region acquired after color filtering
Related candidate region, filters here by the candidate region for obtaining, and filters out and does not meet preset areas
The candidate region of characteristic of field, so as to obtain the candidate region for meeting predeterminable area feature.Predeterminable area is characterized in
The Regional Characteristics that default significant figure region should have.
Discrimination module 1004, after feature is extracted in the candidate region for that will filter out, the feature of extraction is passed through
Grader for discriminating whether to belong to significant graphics class is differentiated, to obtain road traffic sign detection result.
Specifically, grader is by belonging to the positive sample collection of significant graphics class and being not belonging to significant graphics class
Negative sample collection training.The positive sample that wherein positive sample is concentrated is the image for including significant figure;Negative sample
The negative sample of concentration is then not include the image of significant figure, and negative sample can be included with significant figure not
Related random image, it is also possible to including the image similar to significant figure.
During training grader, the negative sample that the positive sample and negative sample that positive sample is concentrated are concentrated extracts spy respectively
After levying, grader is trained according to the feature extracted.Trained grader can be used to predict one it is new
Whether image belongs to significant graphics class, illustrates to detect traffic sign if significant graphics class is belonged to;
Illustrate to be not detected by traffic sign if significant graphics class is not belonging to.Road traffic sign detection result includes
It is no to detect traffic sign, position of the traffic sign for detecting in street view image can also be included.Wherein
The feature of extraction can be using HOG features, SIFT feature or SURF features etc..Grader can be adopted
With SVMs, cascade classifier or artificial nerve network classifier etc..
Above-mentioned road traffic sign detection device 1000, after street view image is got, according to the face of significant figure
Color span obtains candidate region, can so be filtered out from street view image and substantially not meet traffic sign
Color characteristic region.The predeterminable area for meeting significant figure region is filtered out from candidate region
The candidate region of feature, so can further filter out the predeterminable area feature for substantially not meeting traffic sign
Candidate region.By the differentiation of grader, may finally detect in street view image with the presence or absence of significant
Figure, so as to detect whether there is corresponding traffic sign.The traffic sign of automation can so be realized
Detection, without artificial acquisition, efficiency and accuracy all get a promotion.
As shown in figure 11, in one embodiment, road traffic sign detection device 1000 also includes:Cut out mould
Block 1005, for the statistical information of the appearance position according to traffic sign in pre-acquiring street view image, to street
Scape image is cut.
Specifically, pre-acquiring street view image refers to the advance street view image for obtaining, by pre-acquiring streetscape figure
The appearance position of traffic sign carries out statistics and obtains corresponding statistical information as in.The statistical information can reflect
Go out traffic sign and do not appear in which position of street view image, or be likely to appear in street view image which
A little positions.
Such as, usual street view image can include the image of sky and the image of road, the usual position of image of sky
In the upper area of street view image, the image of road is then located at the lower area of street view image, in these regions
Be not in traffic sign, these regions can be cropped.
In the present embodiment, according to traffic sign in pre-acquiring street view image appearance position statistical information come
Street view image is cut, can directly be abandoned in flow front end the region of traffic sign can not possibly occurs,
So as to improve the efficiency of subsequent treatment.
In one embodiment, candidate region acquisition module 1002 be additionally operable to by street view image with traffic sign
In significant figure color span matching pixel value be set to prospect colour, will be with the color value
The unmatched pixel value of scope is set to background colour, obtains binary image.Then the binary image is entered
Row connection domain lookup, so as to obtain the connected domain being made up of the pixel of prospect colour, can correspond to connected domain
Pixel in original background image is used as candidate region.
Wherein prospect colour can be 0, be visually appear as black;Background colour can be 255, visually show
It is white.Candidate region matches with color span, refers to that the candidate region is corresponding in street view image
Pixel value is in the color span.
Connected domain refers to the image district with the adjacent foreground pixel point composition of same pixel value and position in image
Domain.Connection domain lookup is the process that the connected domain in image is found out and marked, also referred to as connected domain analysis.
Connection domain lookup can use Two-Pass (two-pass scan) algorithms and Seed Filling (seed filling) algorithm.
In one embodiment, candidate region acquisition module 1002 is additionally operable to be obtained and traffic from street view image
A kind of candidate region of the pre-set color span matching of the composition color of the significant figure in mark.Should
It is a kind of to constitute the color that color can use area accounting maximum in significant figure.Such as traffic sign is surveyed for interval
During speed mark, color span can be that the color of camera image is green.By one kind composition color
Color span obtain candidate region, efficiency can be improved and retain marginal information.
In one embodiment, candidate region acquisition module 1002 is additionally operable to be obtained and traffic from street view image
At least one set of in a kind of multigroup pre-set color span of composition color of the significant figure in mark
The candidate region matched somebody with somebody.
Specifically, it is contemplated that outdoor optical is larger according to condition difference, the change of color span is also larger, here
In advance for a kind of composition color sets multigroup pre-set color span, as long as certain region hit one of which is pre-
If color span can be just retained as candidate region, if being unsatisfactory for any group of pre-set color value
Scope just filters out the region.
For example, if traffic sign is to test the speed mark in interval, a kind of composition color of significant figure is green
Color, then a kind of multigroup pre-set color span of composition color of the significant figure in traffic sign is specific
Can be as listed below:
First group:50<H<100, S>55,75<V<160.
Second group:50<H<100, S>50, V<70.
3rd group:60<H<90, S>35,170<V<220.
4th group:28<H<100,110<V<178.
5th group:20<H<110,130<S<230,30<V<60.
Wherein H, S and V represent tonal color passage, saturation degree Color Channel and lightness colors passage respectively
Under pixel value.
In the present embodiment, by setting a kind of multigroup pre-set color span for constituting color, it can be ensured that
Detection of the color filtering step to traffic sign has recall rate higher, prevents missing inspection.Wherein recall rate is again
Claim recall ratio, be the traffic sign for detecting quantity account for physical presence traffic sign quantity ratio.
In one embodiment, predeterminable area feature includes:The size range of significant figure region,
At least one in depth-width ratio scope, color cluster classification number and Gradient Features.
The size range of wherein significant figure region refers to the circumscribed rectangle of significant figure region
Width scope and the scope of height.If than traffic sign to test the speed mark in interval, and the streetscape for obtaining
Picture size is 8192 pixels (width) × 4096 pixel (height), then the interval is tested the speed and indicates the chi of region
Very little scope can be:14 pixels<height<231 pixels, 10 pixels<width<600 pixels;Wherein height
Represent that the interval is tested the speed and indicate the height of region, width represents that the interval is tested the speed and indicates the width of region
Degree.
Depth-width ratio scope refers to the model of the ratio of the height and width of the circumscribed rectangle of significant figure region
Enclose.If than traffic sign to test the speed mark in interval, then test the speed the depth-width ratio model of mark region in the interval
Enclosing to be:1.0<height/width<5.2;Wherein height represents that the interval is tested the speed and indicates the height of region
Degree, width represents that the interval is tested the speed and indicates the width of region.
Color cluster classification number refers to be obtained after the pixel value of significant figure region is clustered
Categorical measure.Cluster can use K-means clustering algorithms, can also be using K-modes and other clusters
Algorithm, will not enumerate.
If than traffic sign to test the speed mark in interval, then K-means clustering algorithms can be used, first from obtaining
Arbitrarily K pixel of selection is used as initial cluster center in the candidate region for taking;And for remaining other pixels,
Then according to their pixel value distances with these cluster centres, closest cluster is assigned these to respectively
Cluster representated by center;Then each cluster centre for obtaining new cluster is calculated again;Constantly repeat this mistake
Journey is untill canonical measure function starts convergence.If it is two classes that can finally cluster, retain corresponding time
Favored area;If it is two classes that can not finally cluster, corresponding candidate region is filtered out.Wherein pixel value
Distance can be calculated using Euclidean distance.Canonical measure function can use mean square deviation.
If image is regarded as two-dimensional discrete function, the gradient of the image is the derivation of this two-dimensional discrete function,
Gradient can reflect the edge of image, and Gradient Features can reflect edge feature in image.If than handing over
The logical interval that is masked as is tested the speed mark, because traffic sign includes a large amount of green areas, is easily mixed with green tree leaf
Confuse.But edge is regular, negligible amounts in the significant figure of traffic sign, and the edge of leaf is then
It is mixed and disorderly, a fairly large number of, so will can be just indicated by the way that the Gradient Features of conversion change can be reflected
Property figure and leaf are distinguished.
In above-described embodiment, can be from the multiple of area size, depth-width ratio, colour type and Gradient Features
Dimension is filtered to the candidate region for obtaining, and filters out the candidate region for substantially not meeting predeterminable area feature,
Reservation there may be the candidate region of significant figure, be easy to improve the effect differentiated subsequently through grader
Rate.And the combination of the predeterminable area feature for passing through multiple dimensions, can cause to be carried out subsequently through grader
The efficiency of differentiation is further lifted.
In one embodiment, street view image acquisition module 1001 is additionally operable to obtain horizontal 360-degree panorama streetscape
Image.
Module 1005 is cut out to be additionally operable to according to traffic sign in pre-acquiring horizontal 360-degree panorama street view image
There is the statistical information of position, horizontal 360-degree panorama street view image is cut.
Candidate region acquisition module 1002 is additionally operable under hsv color model, is obtained from the street view image
Take at least one set of in the green multigroup pre-set color span with the significant figure in traffic sign
The candidate region matched somebody with somebody.Candidate region acquisition module 1002 is specifically under hsv color model, detecting water
Put down 360 degree of each Color Channels of panorama street view image in each pixel value whether with traffic sign in significant figure
Green multigroup pre-set color span at least one set of matching.By horizontal 360-degree panorama streetscape figure
The pixel value that matching is detected as in is set to prospect colour, will be detected as in horizontal 360-degree panorama street view image
Unmatched pixel value is set to background colour, obtains binary image.Connected domain is carried out to binary image to look into
Look for, the connected domain that will be found is used as candidate region.
Candidate region screening module 1003 is additionally operable to be filtered out from candidate region and does not meet significant figure place
The candidate region of the size range in region, depth-width ratio scope, color cluster classification number and Gradient Features.
After discrimination module 1004 is additionally operable to the candidate region extraction HOG features that will retain after filtering, will extract
HOG features by for discriminating whether that the SVMs for belonging to significant graphics class is differentiated, to obtain
Obtain road traffic sign detection result.
One of ordinary skill in the art will appreciate that all or part of flow in realizing above-described embodiment method,
Computer program be can be by instruct the hardware of correlation to complete, described program can be stored in a calculating
In machine read/write memory medium, the program is upon execution, it may include such as the flow of the embodiment of above-mentioned each method.
Wherein, described storage medium can for magnetic disc, CD, read-only memory (Read-Only Memory,
The non-volatile memory medium such as ROM), or random access memory (Random Access Memory, RAM)
Deng.
Each technical characteristic of embodiment described above can be combined arbitrarily, not right to make description succinct
The all possible combination of each technical characteristic in above-described embodiment is all described, as long as however, these skills
The combination of art feature does not exist contradiction, is all considered to be the scope of this specification record.
Embodiment described above only expresses several embodiments of the invention, and its description is more specific and detailed,
But can not therefore be construed as limiting the scope of the patent.It should be pointed out that for this area
For those of ordinary skill, without departing from the inventive concept of the premise, some deformations can also be made and changed
Enter, these belong to protection scope of the present invention.Therefore, the protection domain of patent of the present invention should be with appended power
Profit requires to be defined.
Claims (12)
1. a kind of method for traffic sign detection, methods described includes:
Obtain street view image;
Obtain what is matched with the color span of the significant figure in traffic sign from the street view image
Candidate region;
Candidate region to obtaining is screened, and the candidate region for filtering out meets where the significant figure
The predeterminable area feature in region;
After feature is extracted in the candidate region that will be filtered out, by the feature of extraction by for discriminating whether to belong to mark
The grader of will graphics class is differentiated, to obtain road traffic sign detection result.
2. method according to claim 1, it is characterised in that described to be obtained from the street view image
Before the candidate region matched with the color span of the significant figure in traffic sign, also include:
The statistical information of the appearance position according to the traffic sign in pre-acquiring street view image, to the street
Scape image is cut.
3. method according to claim 1, it is characterised in that described to be obtained from the street view image
The candidate region matched with the color span of the significant figure in traffic sign, including:
Obtained from the street view image multigroup with a kind of composition color of the significant figure in traffic sign
The candidate region of at least one set of matching in pre-set color span.
4. method according to claim 1, it is characterised in that described to be obtained from the street view image
The candidate region matched with the color span of the significant figure in traffic sign, including:
By what is matched with the color span of the significant figure in the traffic sign in the street view image
Pixel value is set to prospect colour, will be set to background colour with the unmatched pixel value of color span,
Obtain binary image;
The connected domain of prospect colour composition is searched to get candidate region in the binary image.
5. method according to claim 1, it is characterised in that the street view image is horizontal 360-degree
Panorama street view image;The traffic sign is to test the speed mark in interval;
The color span obtained from the street view image with the significant figure in traffic sign
The candidate region matched somebody with somebody, including:
Under hsv color model, obtained from the street view image and the significant figure in traffic sign
The candidate region of at least one set of matching in multigroup pre-set color span of green;
After feature is extracted in the candidate region that will be filtered out, by the feature of extraction by for discriminating whether category
Differentiated in the grader of significant graphics class, to obtain road traffic sign detection result, including:
After HOG features are extracted in the candidate region that will be filtered out, by the HOG features of extraction by for differentiating
Whether the SVMs for belonging to significant graphics class is differentiated, to obtain road traffic sign detection result.
6. method as claimed in any of claims 1 to 5, it is characterised in that described pair of acquisition
Candidate region screened, the candidate region for filtering out meets the default of the significant figure region
Provincial characteristics, including:
Size range, the Gao Kuan for not meeting significant figure region are filtered out from the candidate region for obtaining
Than the candidate region of any one in scope, color cluster classification number and Gradient Features.
7. a kind of road traffic sign detection device, it is characterised in that described device includes:
Street view image acquisition module, for obtaining street view image;
Candidate region acquisition module, for being obtained from the street view image and the significant figure in traffic sign
The candidate region of the color span matching of shape;
Candidate region screening module, for being screened to the candidate region for obtaining, the candidate region for filtering out
Meet the predeterminable area feature of the significant figure region;
Discrimination module, after feature is extracted in the candidate region for that will filter out, by the feature of extraction by being used for
Discriminate whether that the grader for belonging to significant graphics class is differentiated, to obtain road traffic sign detection result.
8. device according to claim 7, it is characterised in that described device also includes:
Module is cut out, for the statistics of the appearance position according to the traffic sign in pre-acquiring street view image
Information, cuts to the street view image.
9. device according to claim 7, it is characterised in that the candidate region acquisition module is specific
For obtaining multigroup with a kind of composition color of the significant figure in traffic sign from the street view image
The candidate region of at least one set of matching in pre-set color span.
10. device according to claim 7, it is characterised in that the candidate region acquisition module tool
Body is used to match the color span in the street view image with the significant figure in the traffic sign
Pixel value be set to prospect colour, background colour will be set to the unmatched pixel value of color span,
Obtain binary image;The connected domain of prospect colour composition is searched to get candidate in the binary image
Region.
11. devices according to claim 7, it is characterised in that the street view image is level 360
Degree panorama street view image;The traffic sign is to test the speed mark in interval;
The candidate region acquisition module from the street view image specifically under hsv color model, obtaining
Take at least one set of in the green multigroup pre-set color span with the significant figure in traffic sign
The candidate region matched somebody with somebody;
After the discrimination module extracts HOG features specifically for the candidate region that will filter out, by what is extracted
HOG features discriminate whether that the SVMs for belonging to significant graphics class is differentiated by being used for, to obtain
Road traffic sign detection result.
12. device according to any one in claim 7 to 11, it is characterised in that the candidate
Region screening module does not meet significant figure region specifically for being filtered out from the candidate region for obtaining
Size range, depth-width ratio scope, color cluster classification number and Gradient Features in the candidate of any one
Region.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510797278.0A CN106709412B (en) | 2015-11-17 | 2015-11-17 | Traffic sign detection method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510797278.0A CN106709412B (en) | 2015-11-17 | 2015-11-17 | Traffic sign detection method and device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106709412A true CN106709412A (en) | 2017-05-24 |
CN106709412B CN106709412B (en) | 2021-05-11 |
Family
ID=58932413
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201510797278.0A Active CN106709412B (en) | 2015-11-17 | 2015-11-17 | Traffic sign detection method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106709412B (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107392115A (en) * | 2017-06-30 | 2017-11-24 | 中原智慧城市设计研究院有限公司 | A kind of traffic sign recognition method based on layered characteristic extraction |
CN108021868A (en) * | 2017-11-06 | 2018-05-11 | 南京航空航天大学 | A kind of quick highly reliable circular target detection recognition method |
CN109447139A (en) * | 2018-10-18 | 2019-03-08 | 中国农业大学 | Farm's scene point cloud recognition methods and device comprising complete domestic animal |
CN111325970A (en) * | 2018-12-14 | 2020-06-23 | 沈阳美行科技有限公司 | Road speed limit prompting method and device and related equipment |
CN112508989A (en) * | 2020-11-20 | 2021-03-16 | 腾讯科技(深圳)有限公司 | Image processing method, device, server and medium |
CN113191272A (en) * | 2021-04-30 | 2021-07-30 | 杭州品茗安控信息技术股份有限公司 | Engineering image identification method, identification system and related device |
CN113826107A (en) * | 2019-05-23 | 2021-12-21 | 日立安斯泰莫株式会社 | Object recognition device |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102799859A (en) * | 2012-06-20 | 2012-11-28 | 北京交通大学 | Method for identifying traffic sign |
CN103366190A (en) * | 2013-07-26 | 2013-10-23 | 中国科学院自动化研究所 | Method for identifying traffic sign |
CN103390167A (en) * | 2013-07-18 | 2013-11-13 | 奇瑞汽车股份有限公司 | Multi-characteristic layered traffic sign identification method |
CN103544484A (en) * | 2013-10-30 | 2014-01-29 | 广东工业大学 | Traffic sign identification method and system based on SURF |
CN104463105A (en) * | 2014-11-19 | 2015-03-25 | 深圳市腾讯计算机系统有限公司 | Guide board recognizing method and device |
CN104616021A (en) * | 2014-12-24 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Method and device for processing traffic sign symbols |
CN104867192A (en) * | 2015-04-30 | 2015-08-26 | 公安部交通管理科学研究所 | Automobile driving path identification system based on automotive electronic identification, and method thereof |
-
2015
- 2015-11-17 CN CN201510797278.0A patent/CN106709412B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102799859A (en) * | 2012-06-20 | 2012-11-28 | 北京交通大学 | Method for identifying traffic sign |
CN103390167A (en) * | 2013-07-18 | 2013-11-13 | 奇瑞汽车股份有限公司 | Multi-characteristic layered traffic sign identification method |
CN103366190A (en) * | 2013-07-26 | 2013-10-23 | 中国科学院自动化研究所 | Method for identifying traffic sign |
CN103544484A (en) * | 2013-10-30 | 2014-01-29 | 广东工业大学 | Traffic sign identification method and system based on SURF |
CN104463105A (en) * | 2014-11-19 | 2015-03-25 | 深圳市腾讯计算机系统有限公司 | Guide board recognizing method and device |
CN104616021A (en) * | 2014-12-24 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Method and device for processing traffic sign symbols |
CN104867192A (en) * | 2015-04-30 | 2015-08-26 | 公安部交通管理科学研究所 | Automobile driving path identification system based on automotive electronic identification, and method thereof |
Non-Patent Citations (3)
Title |
---|
常发亮等: "基于高斯颜色模型和SVM的交通标志检测", 《仪器仪表学报》 * |
李雷达: "《图像质量评价中的特征提取方法与应用》", 30 June 2015 * |
陈慧岩等: "《无人驾驶汽车概论》", 31 July 2017 * |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107392115A (en) * | 2017-06-30 | 2017-11-24 | 中原智慧城市设计研究院有限公司 | A kind of traffic sign recognition method based on layered characteristic extraction |
CN108021868A (en) * | 2017-11-06 | 2018-05-11 | 南京航空航天大学 | A kind of quick highly reliable circular target detection recognition method |
CN109447139A (en) * | 2018-10-18 | 2019-03-08 | 中国农业大学 | Farm's scene point cloud recognition methods and device comprising complete domestic animal |
CN109447139B (en) * | 2018-10-18 | 2021-03-05 | 中国农业大学 | Method and device for identifying scene point cloud of farm containing complete livestock |
CN111325970A (en) * | 2018-12-14 | 2020-06-23 | 沈阳美行科技有限公司 | Road speed limit prompting method and device and related equipment |
CN113826107A (en) * | 2019-05-23 | 2021-12-21 | 日立安斯泰莫株式会社 | Object recognition device |
CN112508989A (en) * | 2020-11-20 | 2021-03-16 | 腾讯科技(深圳)有限公司 | Image processing method, device, server and medium |
CN112508989B (en) * | 2020-11-20 | 2024-03-01 | 腾讯科技(深圳)有限公司 | Image processing method, device, server and medium |
CN113191272A (en) * | 2021-04-30 | 2021-07-30 | 杭州品茗安控信息技术股份有限公司 | Engineering image identification method, identification system and related device |
Also Published As
Publication number | Publication date |
---|---|
CN106709412B (en) | 2021-05-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106709412A (en) | Traffic sign detection method and apparatus | |
CN103971126B (en) | A kind of traffic sign recognition method and device | |
CN107516077B (en) | Traffic sign information extraction method based on fusion of laser point cloud and image data | |
CN107506760B (en) | Traffic signal detection method and system based on GPS positioning and visual image processing | |
CN105913041B (en) | It is a kind of based on the signal lamp recognition methods demarcated in advance | |
CN102364496B (en) | Method and system for identifying automobile license plates automatically based on image analysis | |
CN102332092B (en) | Flame detection method based on video analysis | |
CN109086687A (en) | The traffic sign recognition method of HOG-MBLBP fusion feature based on PCA dimensionality reduction | |
CN103488978B (en) | License plate positioning method based on gray level jump and character projection interval mode | |
CN105373794A (en) | Vehicle license plate recognition method | |
CN108090459B (en) | Traffic sign detection and identification method suitable for vehicle-mounted vision system | |
CN104050450A (en) | Vehicle license plate recognition method based on video | |
CN108256467B (en) | Traffic sign detection method based on visual attention mechanism and geometric features | |
CN103295013A (en) | Pared area based single-image shadow detection method | |
Li et al. | A vehicle license plate recognition system based on analysis of maximally stable extremal regions | |
CN107704853A (en) | A kind of recognition methods of the traffic lights based on multi-categorizer | |
CN109255279A (en) | A kind of method and system of road traffic sign detection identification | |
CN109858310A (en) | Vehicles and Traffic Signs detection method | |
CN112241692A (en) | Channel foreign matter intelligent detection and classification method based on aerial image super-pixel texture | |
CN112053371A (en) | Water body extraction method and device in remote sensing image | |
CN111666811A (en) | Method and system for extracting traffic sign area in traffic scene image | |
Chowdhury et al. | An adaptive technique for computer vision based vehicles license plate detection system | |
CN107103295A (en) | Optical remote sensing image cloud detection method of optic | |
CN112241691B (en) | Channel ice condition intelligent identification method based on unmanned aerial vehicle inspection and image characteristics | |
CN116503821B (en) | Road identification recognition method and system based on point cloud data and image recognition |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |