CN115170657A - Steel rail identification method and device - Google Patents
Steel rail identification method and device Download PDFInfo
- Publication number
- CN115170657A CN115170657A CN202210709236.7A CN202210709236A CN115170657A CN 115170657 A CN115170657 A CN 115170657A CN 202210709236 A CN202210709236 A CN 202210709236A CN 115170657 A CN115170657 A CN 115170657A
- Authority
- CN
- China
- Prior art keywords
- target position
- image
- selection frame
- movement
- steel rail
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention provides a steel rail identification method and a steel rail identification device, wherein the method comprises the following steps: performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points; based on a plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position; performing curve fitting on each fitting point to obtain a steel rail in the first image; the effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points positioned in the selection frame when the selection frame is positioned at the target positions. The steel rail identification method and the steel rail identification device can solve the problems that in the deep learning implementation process, a plurality of samples are needed, the real-time performance is not high, and serious errors possibly occur in the traditional image processing-based method, so that steel rail identification with high real-time performance and identification accuracy can be realized.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a steel rail identification method and a steel rail identification device.
Background
The active obstacle detection has predictability, and can detect the obstacle in time before the collision and contact of the train and the obstacle, so that a driver is informed to operate the train to brake emergently or take other measures to avoid collision, and the driving safety is ensured. The active obstacle detection method mainly comprises obstacle detection based on images, radar detection and the like. The image-based active obstacle detection method is widely used due to the advantages of low cost, high flexibility, rich target information, convenience in obstacle classification and the like. In general, active obstacle detection based on images uses fixed semantic information in the images as a reference, namely, uses a rail curve as a reference to construct a safety limit, so as to judge whether a detected target is an obstacle. Therefore, identifying the rail in the image is very important for active obstacle detection.
At present, the method for identifying a steel rail in an image mainly comprises a steel rail identification method based on deep learning and a steel rail identification method based on image processing. The steel rail identification method based on deep learning has high identification accuracy under the condition that the number of samples is large enough and the samples are trained sufficiently, but the real-time performance is not high, so that the method is difficult to be used for active obstacle detection with high real-time performance requirement. The existing steel rail identification method based on image processing has large error, and particularly has low identification accuracy rate on the bent rail under the conditions of uneven illumination, camera shake or rotation and the like.
In conclusion, the prior art is difficult to realize the steel rail identification with higher real-time performance and identification accuracy.
Disclosure of Invention
The invention provides a steel rail identification method and a steel rail identification device, which are used for solving the defect that the steel rail identification with high real-time performance and identification accuracy is difficult to realize in the prior art and realizing the steel rail identification with high real-time performance and identification accuracy.
The invention provides a steel rail identification method, which comprises the following steps:
performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points;
based on the plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position;
performing curve fitting on each fitting point to obtain a steel rail in the first image;
the effective target position refers to the target position of which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting point corresponding to the effective target position is determined based on each feature point positioned in the selection frame when the selection frame is positioned at the target position.
According to the rail identification method provided by the invention, the step of acquiring the target position of the selection frame in each movement based on the plurality of feature points comprises the following steps:
under the condition that the first time number of the movement of the first target direction does not reach a time threshold value and the last moving target position of the selection frame is the effective target position, acquiring the moving direction of the movement based on the fitting points corresponding to the last two effective target positions;
moving the selection frame based on the moving direction of the current movement to obtain the target position of the current movement of the selection frame;
wherein, the lower edge of the target position of the current movement of the selection frame and the upper edge of the target position of the first movement of the selection frame are positioned on the same straight line.
According to the rail identification method provided by the invention, the target position of each movement of the selection frame is obtained based on the plurality of feature points, and the method further comprises the following steps:
and under the condition that the first time of movement in the first target direction does not reach the time threshold value and the last moving target position of the selection frame is not the effective target position, moving the selection frame along the first target direction to acquire the target position of the selection frame in the current movement.
According to the steel rail identification method provided by the invention, the line detection is carried out on the first image based on the line segment detector algorithm to obtain a plurality of characteristic points, and the method comprises the following steps:
based on a first number obtained in advance, segmenting the first image according to a second target direction to obtain a first number of sub-images;
and respectively carrying out line detection on each sub-image based on a line segment detector algorithm to obtain the plurality of feature points.
According to the rail identification method provided by the invention, before the segmentation of the first image in the second target direction based on the first number acquired in advance to obtain the first number of sub-images, the method further comprises the following steps:
based on each second quantity, segmenting the second image according to the second target direction, and acquiring errors of curve segments obtained by approximating the target curve in the second image by straight line segments;
and determining a second quantity corresponding to the minimum value of the errors as the first quantity.
According to the steel rail identification method provided by the invention, before the line detection is carried out on the first image based on the line segment detector algorithm to obtain a plurality of characteristic points, the method further comprises the following steps:
acquiring an original image;
and preprocessing the original image to obtain the first image.
The present invention also provides a rail identification device, comprising:
the characteristic detection module is used for carrying out line detection on the first image based on a line segment detector algorithm to obtain a plurality of characteristic points;
the characteristic screening module is used for acquiring the target position of each moving of the selection frame based on the plurality of characteristic points and acquiring a fitting point corresponding to each effective target position;
the curve fitting module is used for performing curve fitting on each fitting point to obtain the steel rail in the first image;
the effective target position refers to the target position of which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting point corresponding to the effective target position is determined based on each feature point positioned in the selection frame when the selection frame is positioned at the target position.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the rail identification method.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a rail identification method as described in any one of the above.
The present invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a method of identifying a rail as defined in any one of the above.
According to the steel rail identification method and device provided by the invention, according to the imaging characteristics that the steel rail is longitudinally continuous, the color texture difference of the rail bottom, the rail waist and the rail surface is large, and the starting point area of the steel rail imaged in the image is relatively fixed, the thought of approximating a curve by multiple straight lines is utilized, the LSD algorithm is adopted to extract the characteristic points of the steel rail, the characteristic points of the same level are averaged, the least square curve fitting method is adopted to perform fitting, the modeling of the steel rail curve in the image is completed, the steel rail in the image is identified, the problems that in the deep learning implementation process, the number of samples is large, the instantaneity is low, and the major errors possibly occur in the traditional image processing-based method can be overcome, so that the steel rail identification with high instantaneity and identification accuracy can be realized.
Drawings
In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow chart of a rail identification method according to the present invention;
FIG. 2 is a flow chart of a line segment detector algorithm in the rail identification method provided by the present invention;
FIG. 3 is a schematic view of a process of searching for a region in the rail identification method according to the present invention;
FIG. 4 is a schematic view of a straight line segment approximating a curved line segment in the rail identification method according to the present invention;
FIG. 5 is a second schematic diagram of the approximate curve segment of the straight line segment in the rail identification method according to the present invention;
FIG. 6 is a second schematic flow chart of the rail identification method provided by the present invention;
FIG. 7 is a schematic diagram of a region searching process in the rail identification method according to the present invention;
FIG. 8 is a schematic diagram of a region search result in the rail identification method according to the present invention;
FIG. 9 is a schematic diagram of the curve fitting result in the rail identification method according to the present invention;
FIG. 10 is a schematic view of the rail identification device according to the present invention;
fig. 11 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the embodiments of the invention, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance, nor order.
The following describes a rail identification method and apparatus provided by the present invention with reference to fig. 1 to 11.
Fig. 1 is a schematic flow chart of a rail identification method provided by the present invention. As shown in fig. 1, an execution subject of the rail identification method provided by the embodiment of the present invention may be a rail identification apparatus, and the method includes: step 101, step 102 and step 103.
Step 101, performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points.
Specifically, the first image is an image that needs to be subjected to rail identification. The steel rail identification method provided by the embodiment of the invention can be used for identifying the steel rail in the first image.
The steel rail is a main component of the rail transit track. Rail transit may include conventional railways (which may include national, inter-city, and urban railways, etc.), subways, light rails, and trams.
The Line Segment Detector (LSD) algorithm is a Line Segment detection algorithm, and can obtain a detection result of sub-pixel level precision in a linear time (linear-time), and obtain a linear detection result of higher precision in a shorter time, and mainly uses the gradient of each pixel point as a basis to detect a linear Segment.
Fig. 2 is a flow chart of a line segment detector algorithm in the rail identification method provided by the invention. The LSD algorithm may be implemented as shown in fig. 2.
Alternatively, the first image may be used as an input, the LSD algorithm shown in fig. 2 is performed on the first image, the first image is subjected to line detection, and each straight line segment in the first image is detected; and taking both end points of each straight-line segment as feature points, so that a plurality of feature points can be obtained.
And 102, acquiring the target position of each movement of the selection frame based on the plurality of feature points, and acquiring a fitting point corresponding to each effective target position.
The effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points positioned in the selection frame when the selection frame is positioned at the target positions.
Specifically, the starting region (which may be a rectangular region) of the steel rail in the first image may be obtained in advance based on the prior characteristics of the acquired images, that is, the prior knowledge, and the starting region is used as the target position of the 0 th movement of the selection frame.
Alternatively, the left rail and the right rail may appear simultaneously in the general image, and thus the number of the selection frames may be two, corresponding to the left rail and the right rail, respectively.
Alternatively, the geometric description of the starting region may be:
start area [ x ] corresponding to left rail l1 ,y l1 ]、[x l2 ,y l2 ]、[x l3 ,y l3 ]、[x l4 ,y l4 ](ii) a Starting region [ x ] corresponding to right track r1 ,y r1 ]、[x r2 ,y r2 ]、[x r3 ,y r3 ]、[x r4 ,y r4 ]。
Wherein, [ x ] l1 ,y l1 ]、[x l2 ,y l2 ]、[x l3 ,y l3 ]、[x l4 ,y l4 ]Coordinates of four vertexes of an initial area corresponding to the left rail respectively; [ x ] of r1 ,y r1 ]、[x r2 ,y r2 ]、[x r3 ,y r3 ]、[x r4 ,y r4 ]Coordinates of four vertexes of the start area corresponding to the right rail respectively。
For each selection box, the following processing may be performed:
and taking the target position of the 0 th movement of the selection frame as a starting point, performing area search based on a preset area search algorithm, sequentially determining the target position of each movement of the selection frame until a preset search termination condition is met, and finishing the area search.
The purpose of performing the area search by moving the selection box is to screen the feature points acquired in step 101, screen out feature points located on the rail, and eliminate other feature points (for example, feature points located on the edge of the pantograph pillar or the track bed).
After determining the target position for each movement of the selection box, the magnitude relationship between the number of feature points located within the selection box when the selection box is located at the target position and the number threshold may be compared.
Note that the feature points located in the selection frame include feature points located on the boundary of the selection frame.
Determining the target position as a valid target position when the number of the feature points in the selection frame is greater than or equal to the number threshold when the selection frame is located at the target position, that is, the target position is a valid target position; in the case where the number of feature points located within the selection box when the selection box is located at the target position is less than the number threshold, the target position is not determined as a valid target position, that is, the target position is not a valid target position.
The number threshold can be set according to actual conditions. The embodiment of the present invention is not particularly limited with respect to the specific value of the number threshold. Illustratively, the quantity threshold may be 3, 4, or 5, etc.
For each valid target position, when the selection frame is located at the target position, the feature points located in the selection frame may be defined as the same level (may be the same horizontal level or the same vertical level), and based on the feature points located in the selection frame when the selection frame is located at the target position, the fitting point corresponding to the valid target position is determined.
Optionally, the coordinates of the fitting point corresponding to the effective target position may be obtained based on the coordinates of each feature point located in the selection frame when the selection frame is located at the target position, so as to obtain the fitting point corresponding to the effective target position.
Alternatively, a mathematical statistic (for example, an average value or a weighted average value) of coordinates of each feature point located in the selection frame when the selection frame is located at the target position may be obtained as coordinates of the fitting point corresponding to the effective target position based on a mathematical statistic method.
For example, when the selection frame is located at the target position, an arithmetic average of coordinates of each feature point located in the selection frame may be obtained as coordinates of the fitting point corresponding to the valid target position.
Wherein x is vk ,y vk Respectively representing the abscissa and the ordinate of the fitting point corresponding to the target position of the kth movement; x is the number of i ,y i The abscissa and the ordinate of the ith characteristic point positioned in the selection frame when the selection frame is positioned at the target position of the kth movement are shown; n represents the number of feature points located in the selection box when the selection box is located at the target position of the k-th movement. It is to be understood that N is typically greater than or equal to the quantity threshold.
It can be understood that, in the case where the target position of the selection box moved one time is not the valid target position, the step of determining the fitting point corresponding to the target position based on each feature point located in the selection box when the selection box is located at the target position is not performed, that is, there is no fitting point corresponding to the target position.
And 103, performing curve fitting on the fitting points to obtain the steel rail in the first image.
Specifically, any curve fitting method can be adopted to perform curve fitting on each fitting point to obtain a curve equation for describing the steel rail in the first image, and the curve equation is used as a steel rail model; based on the curve equation, the rail in the first image can be identified.
Alternatively, a least squares method may be employed to curve fit the fitted points.
Alternatively, a curve fitting may be performed on each fitting point by using a least square method with a cubic polynomial as an objective function.
The method comprises the steps of designing a region search algorithm by fully utilizing the imaging characteristics of the steel rail, fixing a starting point region of the steel rail imaged in the image as a prior condition, moving a search box, and gradually reducing a search range in the process of moving the search box so as to match the characteristics of continuous and smooth steel rail imaging and small distant view imaging range.
In the embodiment of the invention, only the line characteristics of the steel rail in the image are utilized, strict conditions such as illumination and anti-shaking are not needed, when the long-range image is fuzzy in details and cannot be accurately identified, the region search algorithm can be automatically terminated at the long-range scene, model errors caused by searching interference points are avoided, the accuracy of steel rail identification can be ensured, the region search is only an algebraic iteration process, the algorithm is simple and convenient, and the real-time performance is better.
According to the imaging characteristics that the longitudinal continuity of the steel rail, the color texture difference of the rail bottom, the rail waist and the rail surface is large, and the imaging starting point area of the steel rail in the image is relatively fixed, the embodiment of the invention utilizes the thought of a multi-section straight line approximate curve, adopts an LSD algorithm to extract the characteristic points of the steel rail, averages the characteristic points of the same level grade, adopts a least square curve fitting method to fit, completes the modeling of the steel rail curve in the image, identifies the steel rail in the image, can overcome the problems of more samples, low real-time performance, serious errors possibly generated by a traditional image processing-based method in the deep learning implementation process, and the like, and can realize the identification of the steel rail with high real-time performance and identification accuracy.
Based on the content of any of the above embodiments, acquiring the target position of each movement of the selection frame based on the plurality of feature points includes: and under the condition that the first time number of the movement of the first target direction does not reach the time threshold value and the target position of the last movement of the selection frame is an effective target position, acquiring the movement direction of the current movement based on the fitting points corresponding to the two effective target positions.
Specifically, it should be noted that the 1 st movement of the selection box takes the first target direction as the moving direction.
Alternatively, the first target direction may be a width direction or a height direction of the first image.
Preferably, the first target direction may be a height direction of the first image, so as to adapt to a case that the direction of the steel rail in the general image is substantially a bottom-to-top direction.
The target position of the 1 st movement of the selection frame can be obtained by moving the selection frame m pixels in the first target direction with the target position of the 0 th movement of the selection frame as a starting point. Where m may be the height of the selection box.
It should be noted that the target position of the 0 th movement of the selection box is directly determined as the valid target position, and whether the number of feature points located in the selection box is greater than or equal to the number threshold when the selection box is located at the target position of the 0 th movement. The target position of the 0 th movement of the selection frame is the 0 th effective target position, and the fitting point corresponding to the 0 th effective target position is marked as V 0 。
After the selection box moves for the kth time, it is first determined whether the first number of times of movement in the first target direction reaches (means is greater than or equal to) a threshold number of times. If so, stopping moving the search box, and ending the area search; if not, judging whether the target position of the k-th movement of the selection frame is a valid target position.
The first number of times of movement in the first target direction refers to the number of times of continuous movement in the first target direction in the process that the selection frame moves from the last valid target position to the last moved target position.
The threshold value of the times can be set according to actual conditions. The embodiment of the present invention is not particularly limited with respect to the specific value of the number threshold. Illustratively, the number threshold may be 2 or 3, etc.
If the target position of the k-th movement of the selection frame is a valid target position, the fitting point V corresponding to the target position of the k-th movement can be determined i Will fit the point V i As the fitting point corresponding to the last effective target position.
For the (k + 1) th movement of the selection frame, the fitting point corresponding to the upper two effective target positions is V i-1 And V i . Based on the principle that two points determine a straight line, the fitting point V can be determined i-1 Point of approach V i Is determined as the moving direction of the (k + 1) th movement of the selection box.
And moving the selection frame based on the moving direction of the current movement to obtain the target position of the current movement of the selection frame.
Wherein, the lower edge of the target position of the current movement of the selection frame and the upper edge of the target position of the first movement of the selection frame are positioned on the same straight line.
Specifically, for the (k + 1) th movement of the selection frame, the selection frame may be moved along the movement direction of the (k + 1) th movement of the selection frame until the movement distance of the selection frame in the first target direction reaches the m pixels, so that the lower edge of the target position of the (k + 1) th movement of the selection frame and the upper edge of the target position of the (k + 1) th movement of the selection frame are located on a straight line, thereby obtaining the target position of the (k + 1) th movement of the selection frame.
The embodiment of the invention fully utilizes the imaging characteristics of the steel rail, namely the imaging characteristics of continuity, smoothness, remarkable edge characteristics and the like of the steel rail in the image, designs the region search algorithm, fixes the starting point region of the steel rail imaged in the image as the prior condition, gradually and iteratively constructs the prediction straight line, moves the search box according to the direction of the prediction straight line, and gradually reduces the search range in the process of moving the search box so as to match the characteristics of continuity, smoothness and small distant view imaging range of the steel rail imaging, and can obtain more accurate fitting points, thereby obtaining more accurate steel rail search results and having higher real-time property.
Based on the content of any of the above embodiments, obtaining the target position of each movement of the selection frame based on the plurality of feature points, further includes: and under the condition that the first time number of the movement of the first target direction does not reach the time threshold value and the target position of the previous movement of the selection frame is not the effective target position, moving the selection frame along the first target direction to obtain the target position of the current movement of the selection frame.
Specifically, after the selection box moves for the kth time, it is first determined whether the first number of times of movement in the first target direction reaches (means is greater than or equal to) a number of times threshold. If so, stopping moving the search box, and ending the area search; if not, judging whether the target position of the k-th movement of the selection frame is a valid target position.
If the target position of the k-th movement of the selection frame is not the valid target position, the first target direction may be determined as the movement direction of the k + 1-th movement of the selection frame; and moving the selection frame along the first target direction until the moving distance of the selection frame in the first target direction reaches the m pixels, so that the lower edge of the target position of the k +1 th movement of the selection frame and the upper edge of the target position of the k +1 th movement of the selection frame are positioned on a straight line, and the target position of the k +1 th movement of the selection frame is obtained.
Fig. 3 is a schematic flow chart of area search in the rail identification method provided by the present invention. The process of obtaining the target position of each movement of the selection box through the area search may be as shown in fig. 3.
Note that, M in fig. 3 indicates a first number; n represents the number of feature points located within the selection box when the selection box is located at the target position.
The embodiment of the invention fully utilizes the imaging characteristics of the steel rail, namely the imaging characteristics of continuity, smoothness, remarkable edge characteristics and the like of the steel rail in the image, designs the region search algorithm, fixes the starting point region of the steel rail imaged in the image as the prior condition, gradually and iteratively constructs the prediction straight line, moves the search box according to the direction of the prediction straight line, and gradually reduces the search range in the process of moving the search box so as to match the characteristics of continuity, smoothness and small distant view imaging range of the steel rail imaging, and can obtain more accurate fitting points, thereby obtaining more accurate steel rail search results and having higher real-time property.
Based on the content of any of the above embodiments, performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points, including: and based on the first quantity acquired in advance, segmenting the first image according to the second target direction to obtain a first quantity of sub-images.
In particular, the first image may be segmented according to the second target direction, splitting the first image into a first number of sub-images.
Alternatively, the second target direction may be a width direction or a height direction of the first image.
Preferably, the second target direction may be a width direction of the first image, so as to adapt to a case that the direction of the steel rail in the general image is substantially a bottom-to-top direction.
Optionally, the first image may be equally divided according to the second target direction based on the first number acquired in advance to obtain a first number of sub-images.
Alternatively, in the case where the second target direction is the width direction of the first image, the left and right portions of the first image may be subjected to division with a larger scale to obtain a sub-image with a larger width, and the middle portion of the first image may be subjected to division with a smaller scale to obtain a sub-image with a smaller width.
Alternatively, in the case where the second target direction is the height direction of the first image, the segmentation with a larger scale may be performed at the upper part of the first image to obtain a sub-image with a larger height, and the segmentation with a smaller scale may be performed at the middle part and the lower part of the first image to obtain a sub-image with a smaller height.
Alternatively, the first number may be a factor of the width or height of the first image. In the case where the second target direction is a width direction of the first image, the first number may be a factor of a width of the first image; in case the second target direction is a height direction of the first image, the first number may be a factor of the height of the first image.
Illustratively, the size of the first image is Q × P, i.e. the width of the first image is Q pixels and the height is P pixels, the first number may be a factor of Q.
Alternatively, the first number may be 10 to 15.
And respectively carrying out line detection on each sub-image based on a line segment detector algorithm to obtain a plurality of feature points.
Specifically, each sub-image may be used as an input, the LSD algorithm shown in fig. 2 is performed on the sub-image, line detection is performed on the sub-image, and each straight line segment in the sub-image is detected; and taking both end points of each straight-line segment as feature points, so that a plurality of feature points can be obtained.
The union of the feature points acquired from the sub-images may be used as a set of feature points in the first image. And the set of each feature point in the first image is a feature point set to be processed.
According to the embodiment of the invention, a curve segmentation scheme is designed by a method of approximating a curve by a straight line, an LSD algorithm is ingeniously utilized, and based on the characteristic that an intersected straight line in an image is split into a plurality of straight line segments at an intersection point, the segmented image is subjected to line detection, so that more steel rail characteristic points are obtained.
According to the embodiment of the invention, the first image is divided into the plurality of sub-images, and the line detection is respectively carried out on each sub-image, so that the segmented detection length of the steel rail curve can be shortened, more steel rail characteristic points can be obtained, and a more accurate steel rail identification result can be obtained based on more steel rail characteristic points.
Based on the content of any of the above embodiments, before segmenting the first image in the second target direction based on the first number acquired in advance to obtain the first number of sub-images, the method further includes: and segmenting the second image in the second target direction based on the second quantities respectively, and acquiring errors of curve segments into which the target curve in the second image is segmented by using straight line segments.
Specifically, the second image is an image having the same size as the first image.
Optionally, the second image is an image in which a rail has been marked or identified.
Alternatively, the target curve may be a rail in the second image.
For each second number, the second image may be segmented in a second target direction based on the second number, correspondingly segmenting the target curve into a plurality of curve segments.
It can be understood that the method steps for segmenting the second image in the second target direction based on the second number are similar to the method steps for segmenting the first image in the second target direction based on the first number acquired in advance, and are not described herein again.
Fig. 4 is one of the principle schematic diagrams of a straight line segment approximate curve segment in the rail identification method provided by the invention. As shown in fig. 4, the curve segment L can be approximated by a straight line segment L, and the endpoints of the curve segment L and the straight line segment L are points a and B. The error of approximating the curved line segment L with the straight line segment L can be defined as the maximum vertical distance h from a point on the curved line segment L to the straight line segment L. Therefore, h may represent the error of the straight line approximation substitution curve.
Fig. 5 is a second schematic diagram of the principle of the approximate curve segment of the straight line segment in the rail identification method provided by the invention. As shown in fig. 5, in the case where the curved line segment L is divided into 3 segments (curved line segment AC, curved line segment CD, and curved line segment DB), the curved line segment L may pass through 3 straight line segments L 1 -l 2 -l 3 To make an approximation, the curve segment AC is represented by a straight line segment l 1 Approximating, a straight line segment l for a curved segment CD 2 Approximating, for the curved section DB, a straight section l 3 And (4) approximation. Thus, straight line segment l 1 -l 2 -l 3 Error of approximate curve segment AB is Σ h i I =1,2,3; wherein h is 1 Straight line segment l for representing curve segment AC 1 Error of approximation, h 2 Straight line segment l for representing curve segment CD 2 Error of approximation, h 3 Straight line segment l for representing curve segment DB 3 Error of approximation.
For any second number u, the target curve in the second image is divided into v curve segments (v ≦ u), and the error of the curve segment into which the target curve in the second image is divided is ∑ h i ，i＝1,2,3,…,v。
And determining a second quantity corresponding to the minimum value of the errors as the first quantity.
Specifically, among the second numbers, the error of approximating the curve segment into which the target curve is divided in the second image by a certain second number of straight-line segments is smallest, which means that the effect of approximating the target curve by the second number of straight-line segments is the best as the second number of straight-line segments is closer to the target curve, the second number is the second number corresponding to the minimum value of the error, the second number may be set as the optimal division number, and the second number may be determined as the first number.
According to the embodiment of the invention, an optimal curve segmentation scheme can be designed by determining the optimal first quantity.
According to the embodiment of the invention, the error of the curve segment obtained by dividing the second image according to each second quantity and obtained by approximating the target curve in the second image by the straight line segment is determined as the first quantity according to the second quantity corresponding to the minimum value of the error, so that the quantity and the accuracy of the obtained characteristic points can be improved, and a more accurate steel rail identification result can be obtained.
Based on the content of any of the above embodiments, before performing line detection on the first image based on the line segment detector algorithm to obtain a plurality of feature points, the method further includes: an original image is acquired.
Specifically, the original image may be acquired based on an image or video of the target road segment acquired by the image acquisition device.
The image acquisition device can be a camera or a video camera and the like.
The image acquisition device can acquire the original image by shooting the target road section or the like, or acquire the video of the target road section by shooting or the like.
The video of the target road section can be processed by screenshot and the like, and an original image is obtained.
And preprocessing the original image to obtain a first image.
Specifically, the original image may be preprocessed to obtain the first image.
The pretreatment method can comprise at least one of the following steps: graying, noise filtering, and inverse filtering (i.e., deblurring).
According to the embodiment of the invention, the original image is preprocessed to obtain the first image, interference factors such as noise in the original image are removed, and the accuracy of the steel rail identification result obtained by identifying the steel rail on the first image is higher.
In order to facilitate understanding of the above embodiments of the present invention, the following describes an implementation process of the rail identification method provided by the above embodiments of the present invention by way of an example.
Fig. 6 is a second schematic flow chart of the rail identification method provided by the present invention. As shown in fig. 6, the following steps may be performed for each frame of image in the video: image preprocessing, image segmentation, feature detection, feature screening, curve fitting and model updating.
And image preprocessing, namely preprocessing the original image to obtain a first image. Theoretical support for the image pre-processing step may include graying and filtering, etc.
And (3) image segmentation, namely equally dividing the first image into a first number of sub-images. Theoretical support for the image segmentation step may include optimal equal-division segmentation, etc. The first number, i.e. the predetermined optimal number of equal parts.
And (4) feature detection, namely acquiring feature points in the first image based on an LSD algorithm. Theoretical support for the feature detection step may include the LSD algorithm, etc.
And (4) feature screening, namely performing region search based on a region search algorithm to obtain each fitting point. Theoretical support for the feature screening step may include area searching, etc.
And (4) curve fitting, namely performing curve fitting on each fitting point obtained in the characteristic screening step to obtain a steel rail model. Theoretical support for curve fitting may include least squares, and the like.
And (4) updating the model, namely updating the model based on the steel rail model obtained in the curve fitting step.
FIG. 7 is a schematic diagram of a region searching process in the rail identification method according to the present invention; FIG. 8 is a schematic diagram of a region search result in the rail identification method according to the present invention; fig. 9 is a schematic diagram of a curve fitting result in the rail identification method provided by the present invention.
As shown in fig. 7 to 9, in the process of performing the rail identification method shown in fig. 6 on a certain original image, the region search process may be as shown in fig. 7, the region search result may be as shown in fig. 8, and the curve fitting result (i.e., the identification result of the left and right rails) may be as shown in fig. 9.
The rail identification device provided by the invention is described below, and the rail identification device described below and the rail identification method described above can be referred to correspondingly.
Fig. 10 is a schematic structural view of a rail identifying apparatus provided by the present invention. Based on the content of any of the above embodiments, as shown in fig. 10, the apparatus includes a feature detection module 1001, a feature screening module 1002, and a curve fitting module 1003, wherein:
a feature detection module 1001, configured to perform line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points;
the feature screening module 1002 is configured to obtain a target position of each movement of the selection box based on the plurality of feature points, and obtain a fitting point corresponding to each effective target position;
a curve fitting module 1003, configured to perform curve fitting on each fitting point to obtain a steel rail in the first image;
the effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points located in the selection frame when the selection frame is located at the target position.
Specifically, the feature detection module 1001, the feature screening module 1002, and the curve fitting module 1003 may be electrically connected in sequence.
The feature detection module 1001 may execute an LSD algorithm to perform line detection on the first image, and detect each line segment in the first image; and taking both end points of each straight-line segment as feature points, so that a plurality of feature points can be obtained.
The feature filtering module 1002 may perform the following for each selection box:
and taking the target position of the 0 th movement of the selection frame as a starting point, performing area search based on a preset area search algorithm, sequentially determining the target position of each movement of the selection frame until a preset search termination condition is met, and finishing the area search.
The curve fitting module 1003 may perform curve fitting on each fitting point by using any curve fitting method to obtain a curve equation for describing the steel rail in the first image, and use the curve equation as the steel rail model; based on the curve equation, the rail in the first image can be identified.
Optionally, the feature filtering module 1002 may be specifically configured to, when the first number of times of movement in the first target direction does not reach the frequency threshold and the target position of the selection frame that is moved last time is an effective target position, obtain a moving direction of the current movement based on fitting points corresponding to the last two effective target positions; moving the selection frame based on the moving direction of the current movement to obtain the target position of the current movement of the selection frame;
wherein, the lower edge of the target position of the current movement of the selection frame and the upper edge of the target position of the first movement of the selection frame are positioned on the same straight line.
Optionally, the feature screening module 1002 may be further specifically configured to, when the first number of times of movement in the first target direction does not reach the frequency threshold and the target position of the selection box moved last time is not the valid target position, move the selection box along the first target direction to obtain the target position of the selection box moved this time.
Optionally, the feature detection module 1001 may be specifically configured to segment the first image according to the second target direction based on a first number obtained in advance, so as to obtain a first number of sub-images; and respectively carrying out line detection on each sub-image based on a line segment detector algorithm to obtain a plurality of feature points.
Optionally, the rail identification apparatus may further include:
the quantity determining module is used for segmenting the second image according to the second target direction based on each second quantity respectively to obtain errors of curve segments obtained by approximating the straight segments to the segmented target curve in the second image; and determining a second quantity corresponding to the minimum value of the errors as the first quantity.
Optionally, the rail identification apparatus may further include:
the image preprocessing module is used for acquiring an original image; and preprocessing the original image to obtain a first image.
The steel rail identification device provided by the embodiment of the invention is used for executing the steel rail identification method provided by the invention, and the implementation mode of the steel rail identification device is consistent with that of the steel rail identification method provided by the invention, and the same beneficial effects can be achieved, and the details are not repeated here.
The rail identification device is used in the rail identification method of each of the foregoing embodiments. Therefore, the description and definition in the rail identification method in the foregoing embodiments can be used for understanding the execution modules in the embodiments of the present invention.
According to the imaging characteristics that the longitudinal continuity of the steel rail, the color texture difference of the rail bottom, the rail waist and the rail surface is large, and the imaging starting point area of the steel rail in the image is relatively fixed, the embodiment of the invention utilizes the thought of a multi-section straight line approximate curve, adopts an LSD algorithm to extract the characteristic points of the steel rail, averages the characteristic points of the same level grade, adopts a least square curve fitting method to fit, completes the modeling of the steel rail curve in the image, identifies the steel rail in the image, can overcome the problems of more samples, low real-time performance, serious errors possibly generated by a traditional image processing-based method in the deep learning implementation process, and the like, and can realize the identification of the steel rail with high real-time performance and identification accuracy.
Fig. 11 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 11, the electronic device may include: a processor (processor) 1110, a communication Interface (Communications Interface) 1120, a memory (memory) 1130, and a communication bus 1140, wherein the processor 1110, the communication Interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. Processor 1110 may invoke logic instructions in memory 1130 to perform a rail identification method comprising: performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points; based on a plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position; performing curve fitting on each fitting point to obtain a steel rail in the first image; the effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points positioned in the selection frame when the selection frame is positioned at the target positions.
In addition, the logic instructions in the memory 1130 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The processor 1110 in the electronic device provided in the embodiment of the present application may call the logic instruction in the memory 1130, and an implementation manner of the processor is consistent with an implementation manner of the rail identification method provided in the present application, and the same beneficial effects may be achieved, and details are not described here.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, enable the computer to perform the rail identification method provided by the above methods, the method comprising: performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points; based on a plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position; performing curve fitting on each fitting point to obtain a steel rail in the first image; the effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points located in the selection frame when the selection frame is located at the target position.
When the computer program product provided in the embodiment of the present application is executed, the method for identifying a steel rail is implemented, and a specific implementation manner of the method is consistent with the implementation manner described in the embodiment of the foregoing method, and the same beneficial effects can be achieved, which is not described herein again.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, is implemented to perform the method for identifying a rail provided above, the method comprising: performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points; based on a plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position; performing curve fitting on each fitting point to obtain a steel rail in the first image; the effective target position refers to a target position at which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting points corresponding to the effective target positions are determined based on the feature points positioned in the selection frame when the selection frame is positioned at the target positions.
When the computer program stored on the non-transitory computer-readable storage medium provided in the embodiment of the present application is executed, the method for identifying a steel rail is implemented, and the specific implementation manner of the method is consistent with the implementation manner described in the embodiment of the foregoing method, and the same beneficial effects can be achieved, which is not described herein again.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A rail identification method, comprising:
performing line detection on the first image based on a line segment detector algorithm to obtain a plurality of feature points;
based on the plurality of feature points, acquiring the target position of each movement of the selection frame, and acquiring a fitting point corresponding to each effective target position;
performing curve fitting on each fitting point to obtain a steel rail in the first image;
the effective target position refers to the target position of which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting point corresponding to the effective target position is determined based on each feature point positioned in the selection frame when the selection frame is positioned at the target position.
2. The rail identification method according to claim 1, wherein the obtaining of the target position of each movement of the selection frame based on the plurality of feature points includes:
under the condition that the first time number of the movement of the first target direction does not reach a time threshold value and the last moving target position of the selection frame is the effective target position, acquiring the moving direction of the movement based on the fitting points corresponding to the last two effective target positions;
moving the selection frame based on the moving direction of the current movement to obtain the target position of the current movement of the selection frame;
wherein, the lower edge of the target position of the current movement of the selection frame and the upper edge of the target position of the first movement of the selection frame are positioned on the same straight line.
3. The rail identification method according to claim 2, wherein the obtaining of the target position of each movement of the selection frame based on the plurality of feature points further comprises:
and under the condition that the first time of movement in the first target direction does not reach the time threshold value and the last moving target position of the selection frame is not the effective target position, moving the selection frame along the first target direction to acquire the target position of the selection frame in the current movement.
4. The method of claim 1, wherein the line-based detector algorithm performs line detection on the first image to obtain a plurality of feature points, and comprises:
based on a first number obtained in advance, segmenting the first image according to a second target direction to obtain a first number of sub-images;
and respectively carrying out line detection on each sub-image based on a line segment detector algorithm to obtain the plurality of feature points.
5. The method for identifying a steel rail according to claim 4, wherein before the step of segmenting the first image in the second target direction based on the first number acquired in advance to obtain the first number of sub-images, the method further comprises:
based on each second quantity, segmenting the second image according to the second target direction, and acquiring errors of curve segments obtained by approximating the target curve in the second image by straight line segments;
and determining a second quantity corresponding to the minimum value of the errors as the first quantity.
6. The method for identifying a steel rail according to any one of claims 1 to 5, wherein before the line-based detector algorithm performs line detection on the first image to obtain a plurality of feature points, the method further comprises:
acquiring an original image;
and preprocessing the original image to obtain the first image.
7. A rail identification device, comprising:
the characteristic detection module is used for carrying out line detection on the first image based on a line segment detector algorithm to obtain a plurality of characteristic points;
the characteristic screening module is used for acquiring the target position of each moving of the selection frame based on the plurality of characteristic points and acquiring a fitting point corresponding to each effective target position;
the curve fitting module is used for performing curve fitting on the fitting points to obtain the steel rail in the first image;
the effective target position refers to the target position of which the number of the feature points in the selection frame is greater than or equal to a number threshold when the selection frame is located at the target position; the fitting point corresponding to the effective target position is determined based on each feature point positioned in the selection frame when the selection frame is positioned at the target position.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the rail identification method according to any one of claims 1 to 6.
9. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the rail identification method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements a rail identification method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210709236.7A CN115170657A (en) | 2022-06-21 | 2022-06-21 | Steel rail identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210709236.7A CN115170657A (en) | 2022-06-21 | 2022-06-21 | Steel rail identification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115170657A true CN115170657A (en) | 2022-10-11 |
Family
ID=83486894
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210709236.7A Pending CN115170657A (en) | 2022-06-21 | 2022-06-21 | Steel rail identification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115170657A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023176073A1 (en) * | 2022-03-17 | 2023-09-21 | 株式会社日立製作所 | Forward monitoring system and method therefor |
-
2022
- 2022-06-21 CN CN202210709236.7A patent/CN115170657A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023176073A1 (en) * | 2022-03-17 | 2023-09-21 | 株式会社日立製作所 | Forward monitoring system and method therefor |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569704B (en) | Multi-strategy self-adaptive lane line detection method based on stereoscopic vision | |
| CN109784344B (en) | Image non-target filtering method for ground plane identification recognition | |
| CN106778668B (en) | A kind of method for detecting lane lines of robust that combining RANSAC and CNN | |
| US20150279021A1 (en) | Video object tracking in traffic monitoring | |
| CN110287884B (en) | Voltage line detection method in auxiliary driving | |
| CN110197494B (en) | Pantograph contact point real-time detection algorithm based on monocular infrared image | |
| US20230005278A1 (en) | Lane extraction method using projection transformation of three-dimensional point cloud map | |
| CN113988112B (en) | Method, device and equipment for detecting lane line and storage medium | |
| US10249046B2 (en) | Method and apparatus for object tracking and segmentation via background tracking | |
| CN114529459A (en) | Method, system and medium for enhancing image edge | |
| KR20110046607A (en) | Lane detection method and Detecting system using the same | |
| CN114972333B (en) | Road crack detection method and system based on artificial intelligence | |
| CN112767358A (en) | Railway electric locomotive fault detection method based on image feature registration | |
| CN115170657A (en) | Steel rail identification method and device | |
| CN115272306A (en) | Solar cell panel grid line enhancement method utilizing gradient operation | |
| CN113239733B (en) | Multi-lane line detection method | |
| CN112801021B (en) | Method and system for detecting lane line based on multi-level semantic information | |
| CN114529590A (en) | Image registration method and system for motor train unit vehicle bottom fault detection | |
| CN112069924A (en) | Lane line detection method, lane line detection device and computer-readable storage medium | |
| CN111179289B (en) | Image segmentation method suitable for webpage length graph and width graph | |
| CN112033419A (en) | Method, electronic device, and medium for detecting automatic port driving lane line | |
| CN114529715B (en) | Image identification method and system based on edge extraction | |
| CN111126248A (en) | Method and device for identifying shielded vehicle | |
| CN111428538A (en) | Lane line extraction method, device and equipment | |
| CN113284158B (en) | Image edge extraction method and system based on structural constraint clustering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |