CN117830908A - Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene - Google Patents

Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene Download PDF

Info

Publication number
CN117830908A
CN117830908A CN202410030225.5A CN202410030225A CN117830908A CN 117830908 A CN117830908 A CN 117830908A CN 202410030225 A CN202410030225 A CN 202410030225A CN 117830908 A CN117830908 A CN 117830908A
Authority
CN
China
Prior art keywords
drill rod
drill
counting
drilling
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410030225.5A
Other languages
Chinese (zh)
Other versions
CN117830908B (en
Inventor
张富凯
赵珊
张海燕
王登科
闫江伟
司马海峰
田永超
陈旭
陈立伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Henan University of Technology
Original Assignee
Henan University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Henan University of Technology filed Critical Henan University of Technology
Priority to CN202410030225.5A priority Critical patent/CN117830908B/en
Publication of CN117830908A publication Critical patent/CN117830908A/en
Application granted granted Critical
Publication of CN117830908B publication Critical patent/CN117830908B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/41Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0004Industrial image inspection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/46Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/46Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30242Counting objects in image

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Quality & Reliability (AREA)
  • Earth Drilling (AREA)

Abstract

The application discloses a method, a device, equipment and a medium for counting coal mine drill pipes of a self-adaptive scene, which relate to the technical field of artificial intelligent deep learning and comprise the steps of processing and marking historical underground drilling videos to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data; acquiring a current drilling video, and identifying the current drilling video by using a rotating target detection model to obtain an identification result; based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a drill rod selection principle, a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined; and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram. According to the method and the device, the counting of the coal mine drill rods in the self-adaptive scene can be realized, and the accuracy and the efficiency of the counting of the drill rods are improved.

Description

Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene
Technical Field
The invention relates to the technical field of artificial intelligence deep learning, in particular to a method, a device, equipment and a medium for counting coal mine drill pipes in a self-adaptive scene.
Background
Most of the existing technologies of counting drill rods are visual (i.e. target detection) based drill rod counting methods, which use target detection (such as Yolov 5) +deep (object tracking) algorithm, and the identified bounding box is a conventional rectangular box, and the specific counting method is as follows: (1) counting is performed by means of two auxiliary "buffers". (2) The counting method is that the center point or a special point of the drill rod is in the process of going back and forth (and tracking the center point), the drill is returned to count +1 when the center point touches the left buffer zone, and the drill is fed to count +1 when the center point touches the right buffer zone, and the counting is repeated. However, this counting method has problems: (1) the thought is simple, only the collision between the center point of the drill rod and the buffer zone is observed, and a mathematical formula and a mathematical logic are omitted; (2) two buffer areas are required to be drawn in advance, the code is required to be modified when the buffer areas are drawn, the movement position of the center point of the drill rod is required to be observed, and the code is modified according to the movement position range, so that the position of the buffer areas is modified; when the position drawing of the buffer is inaccurate, it is certainly not helpful for counting; (3) the biggest problem is that the position of the camera and the drill must be fixed because of the assistance of the buffer zone, so that the counting is accurate, but the positions of the actual camera and the drill rod can be changed at any time, and when the camera is slightly moved, the buffer zone needs to be redrawn, so that the counting accuracy is affected.
From the above, how to realize the coal mine drill rod counting of the self-adaptive scene, increase the diversity of scene application and the applicability of the drill rod counting in practical application, and improve the accuracy and efficiency of the drill rod counting is a problem to be solved in the field.
Disclosure of Invention
In view of the above, the invention aims to provide a coal mine drill rod counting method, device, equipment and medium for a self-adaptive scene, which can realize the coal mine drill rod counting of the self-adaptive scene, increase the diversity of scene application and the applicability of drill rod counting in practical application, and improve the accuracy and efficiency of drill rod counting. The specific scheme is as follows:
in a first aspect, the application discloses a coal mine drill rod counting method of a self-adaptive scene, which comprises the following steps:
acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for a self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data;
acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result;
based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined;
And calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
Optionally, the processing and labeling the historical downhole drilling video to obtain the drilling data after labeling includes:
dividing the historical underground drilling video into pictures, and cleaning each picture to obtain the cleaned picture;
and marking the picture after the cleaning treatment by using a preset rotating target detection marking tool so as to obtain marking drilling data after marking.
Optionally, the constructing a rotation target detection model for the adaptive scene based on the rotation target detection algorithm and using the noted drilling data includes:
training, testing and verifying a preset initial rotation target detection model based on a rotation target detection algorithm and by using the marked drilling data, so as to obtain the rotation target detection model for the self-adaptive scene; the rotation target detection algorithm comprises Yolov5_OBB, yolov7_OBB, yolov8_ OBB, GGHL, PP-Yolove-R, R3Det and MMRotate algorithm.
Optionally, the identifying the current drilling video by using the rotation target detection model to obtain an identification result includes:
identifying the current drilling video by using the rotating target detection model to obtain the identification result with the rotating boundary box; the identification result comprises a drilling machine whole, a percussion power head, a drilling machine drill rod and corresponding center point coordinates, short side lengths, long side lengths and rotation angles.
Optionally, the drill rod screening is performed on the identification result based on a drill rod counting reasoning algorithm according to a preset drill rod selection principle to obtain a target drill rod, which includes:
constructing the drill rod selection principle; the drill rod selection principle comprises a rotation boundary frame selection principle, a shielding exclusion selection principle and a focused drill rod selection principle;
and based on a drill rod counting reasoning algorithm, and according to the rotating boundary box selection principle, the shielding exclusion selection principle and the focused drill rod selection principle, drill rod screening is carried out on the identification result with the rotating boundary box so as to obtain a target drill rod.
Optionally, the determining the target drill rod key point and the impact power head center point includes:
Determining the current state of the target drill rod; the state comprises drill feeding and drill returning;
and determining corresponding target drill rod key points and impact power head center points based on the states.
Optionally, the calculating, in real time, a distance between the target drill rod key point and the impact power head center point, drawing a peak change chart according to the distance, and counting drill rods by using the peak change chart includes:
if the state is drilling, calculating the drilling distance between the key point of the target drill rod and the central point of the impact power head according to a preset drilling distance calculation method, drawing a drilling peak value change graph according to the drilling distance, and counting the drill rods by using the drilling peak value change graph;
if the state is drill withdrawal, calculating the drill withdrawal distance between the key point of the target drill rod and the central point of the impact power head according to a preset drill withdrawal distance calculation method, drawing a drill withdrawal peak value change graph according to the drill withdrawal distance, and counting the drill rods by using the drill withdrawal peak value change graph.
In a second aspect, the application discloses a colliery drilling rod counting assembly of self-adaptation scene, includes:
the model construction module is used for acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data;
The identification module is used for acquiring a current drilling video, and identifying the current drilling video by utilizing the rotating target detection model so as to obtain an identification result;
the drill rod screening module is used for screening the drill rods according to the identification result based on a drill rod counting reasoning algorithm and a preset drill rod selection principle so as to obtain a target drill rod and determine a target drill rod key point and an impact power head center point;
and the drill rod counting module is used for calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
In a third aspect, the present application discloses an electronic device comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the coal mine drill rod counting method of the self-adaptive scene.
In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; the method comprises the steps of a coal mine drill rod counting method of the self-adaptive scene disclosed above, wherein the steps of the coal mine drill rod counting method of the self-adaptive scene are realized when the computer program is executed by a processor.
It can be seen that the application provides a coal mine drill rod counting method of a self-adaptive scene, which comprises the steps of obtaining a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data; acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result; based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined; and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram. According to the method, the rotating target detection model for the self-adaptive scene is constructed based on the rotating target detection algorithm and by utilizing the drilling data after marking, the current drilling video is identified by utilizing the rotating target detection model, the target drill rod is screened out based on the drill rod counting reasoning algorithm, the peak value change is accurately obtained through calculating the distance change between the key point of the target drill rod and the central point of the impact power head, and the drill rod counting is carried out.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a coal mine drill rod counting method of an adaptive scenario disclosed in the present application;
FIG. 2 is a diagram of an example of drill rod screening and drill rod integrity during drilling as disclosed herein;
FIG. 3 is a diagram of an example of drill rod screening and drill rod integrity during drill withdrawal as disclosed herein;
FIG. 4 is an exemplary graph of recognition results disclosed herein;
FIGS. 5 (a) - (b) are exemplary diagrams of target drill pipe keypoints during drilling as disclosed herein;
FIG. 6 is a graph showing four angular coordinate distribution examples of a target drill rod disclosed in the present application;
FIGS. 7 (a) - (b) are exemplary diagrams of key points of a target drill rod during drill withdrawal disclosed in the present application;
FIG. 8 is a graph illustrating an exemplary peak variation disclosed herein;
FIG. 9 is a flow chart of a coal mine drill pipe counting method for an adaptive scenario disclosed in the present application;
FIG. 10 is a specific flow chart of a drill rod counting disclosed herein;
FIG. 11 is a schematic structural diagram of a coal mine drill rod counting device in an adaptive scenario disclosed in the present application;
fig. 12 is a block diagram of an electronic device provided in the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Most of the existing technologies of counting drill rods are visual (i.e. target detection) based drill rod counting methods, which use target detection (such as Yolov 5) +deep (object tracking) algorithm, and the identified bounding box is a conventional rectangular box, and the specific counting method is as follows: (1) counting is performed by means of two auxiliary "buffers". (2) The counting method is that the center point or a special point of the drill rod is in the process of going back and forth (and tracking the center point), the drill is returned to count +1 when the center point touches the left buffer zone, and the drill is fed to count +1 when the center point touches the right buffer zone, and the counting is repeated. However, this counting method has problems: (1) the thought is simple, only the collision between the center point of the drill rod and the buffer zone is observed, and a mathematical formula and a mathematical logic are omitted; (2) two blue buffer areas are required to be drawn in advance, codes are required to be modified when the buffer areas are drawn, the positions of the drill rod center points are required to be observed, and the codes are modified according to the range of the moving positions, so that the positions of the buffer areas are modified; when the position drawing of the buffer is inaccurate, it is certainly not helpful for counting; (3) the biggest problem is that the position of the camera and the drill must be fixed because of the assistance of the buffer zone, so that the counting is accurate, but the positions of the actual camera and the drill rod can be changed at any time, and when the camera is slightly moved, the buffer zone needs to be redrawn, so that the counting accuracy is affected. From the above, how to realize the coal mine drill rod counting of the self-adaptive scene, increase the diversity of scene application and the applicability of the drill rod counting in practical application, and improve the accuracy and efficiency of the drill rod counting is a problem to be solved in the field.
Referring to fig. 1, the embodiment of the invention discloses a coal mine drill rod counting method in a self-adaptive scene, which specifically comprises the following steps:
step S11: and acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data.
Step S12: and acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result.
In this embodiment, a current drilling video is obtained, and the current drilling video is identified by using the rotation target detection model, so as to obtain the identification result with a rotation boundary box; the identification result comprises a drilling machine whole, a percussion power head, a drilling machine drill rod and corresponding center point coordinates, short side lengths, long side lengths and rotation angles.
Specifically, a current drilling video on site is detected and identified by using a rotating target detection model, and the identification result with a rotating boundary box comprising drilling machine information is identified in real time, wherein the identification information comprises 4 objects of a drilling machine whole, a percussion power head, a drilling machine head and a drilling machine drill rod; and simultaneously obtaining the coordinates of the center points, the lengths of the short sides, the lengths of the long sides and the rotation angles of the 4 objects.
Step S13: and screening the drill rods according to the identification result based on a drill rod counting reasoning algorithm and according to a preset drill rod selection principle to obtain a target drill rod, and determining a target drill rod key point and an impact power head center point.
In this embodiment, the drill rod selection principle is constructed; the drill rod selection principle comprises a rotation boundary frame selection principle, a shielding exclusion selection principle and a focused drill rod selection principle, drill rod screening is carried out on the identification result with the rotation boundary frame based on a drill rod counting reasoning algorithm according to the rotation boundary frame selection principle, the shielding exclusion selection principle and the focused drill rod selection principle so as to obtain a target drill rod, and the current state of the target drill rod is determined; the states include drill-in and drill-out, and corresponding target drill rod key points and impact power head center points are determined based on the states.
In this embodiment, as shown in fig. 2, drill rod screening is performed, a center point coordinate of a whole body (bounding box in fig. 2) of a drilling machine, and four bounding boxes, namely, upper, lower, left, right and left, are obtained, for the left and right boundaries, x coordinates are selected as references, x_min is taken as the left side, and x_max is taken as the right side; for the upper and lower boundaries, selecting a y coordinate as a reference, taking the lower side as y_min, and the upper side as y_max; then, the coordinates of the central point of the impact power head are obtained, and a dot (such as a large dot on the right side in fig. 2) is drawn; the target drill pipe is then determined. The method and the device determine the target drill rod according to a preset drill rod selection principle, and a rotation target detection algorithm can identify all drill rods in a video image, but the method and the device only care about the drill rods which are helpful for counting, other drill rods can be filtered out, and the large round dots on the left side in fig. 2 are key points of the target drill rod.
The drill rod selection principle of the target drill rod is as follows: (1) all drill rods outside the whole body frame of the drilling machine are removed, such as drill rods outside the frame of fig. 3 (many drill rods are often placed on the ground, the drill rods on the ground are not obvious in the picture of fig. 3, but the drill rods on the ground) need to be removed; (2) the drill pipe between the impact head and the drill head (where the drill head is not fully exposed with the blind as shown in fig. 3); (3) the drill rod connected with the outer side of the impact power head is selected as the drill rod which is focused by the invention, such as the drill rod filled in the figures 2 and 3. The whole of the drilling machine, the drill rod, the impact power head and the specific positions of the head of the drilling machine are shown in fig. 4.
The determination of the key point of the target drill rod is divided into two cases of drilling and drilling withdrawal. The drilling situation is shown in fig. 2, the drilling withdrawal situation is shown in fig. 3, (1) the key points of the target drill rod are determined during drilling: the point on the target drill rod furthest from the impact power head (shown in fig. 4) is found when drilling and is used as the key point of the target drill rod. Fig. 2 shows 2 cases, namely (1) the target drill pipe is on the left side of the impact power head, as shown in fig. 5 (a); (2) the standard drill stem is to the right of the impact power head as shown in fig. 5 (b). The specific description is as follows: (1) the position of the target drill rod is at the left side of the impact power head, coordinates of four corners of the target drill rod are obtained, and the ordinate values of the two leftmost points are selected to obtain intermediate points, namely (y1+y4)/2, namely the key points of the target drill rod. The difference in the abscissa distance between the target drill pipe key point and the impact head center point (i.e., the difference between the left dot and the right dot abscissa x in fig. 5 (a)) is calculated as dist. (2) The position of the target drill rod is on the right of the impact power head, coordinates of four corners of the target drill rod are obtained, and the ordinate values of the two rightmost points are selected to obtain intermediate points, namely (y2+y3)/2, namely the key points of the target drill rod. The difference in the abscissa distance between the target drill pipe key point and the impact head center point (i.e., the difference in the right dot and left dot abscissa x in fig. 5 (b)) is calculated as dist. Four point coordinates of y1, y2, y3, y4 are shown in fig. 6.
(2) Determining key points of a target drill rod during drill withdrawal: and (3) finding the nearest point on the target drill rod from the impact power head (shown in fig. 4) during the drill withdrawal, and taking the nearest point as a key point of the target drill rod. Fig. 3 also has 2 cases, namely (1) the target drill pipe is on the left of the impact power head, as shown in fig. 7 (a); (2) the standard drill stem is to the right of the impact power head as shown in fig. 7 (b). The specific description is as follows: (1) the position of the target drill rod is at the left side of the impact power head, coordinates of four corners of the target drill rod are obtained, the ordinate values of the two rightmost points are selected to obtain intermediate points, namely (y2+y3)/2, and the intermediate points are the key points of the target drill rod. The difference in the abscissa distance between the target drill pipe key point and the impact head center point (i.e., the difference between the left dot and the right dot abscissa x in fig. 7 (a)) is calculated as dist. (2) The position of the target drill rod is on the right of the impact power head, coordinates of four corners of the target drill rod are obtained, and the ordinate values of the two leftmost points are selected to obtain intermediate points, namely (y1+y4)/2, namely the key points of the target drill rod. The difference in the abscissa distance between the target drill pipe key point and the impact head center point (i.e., the difference in the right dot and left dot abscissa x in fig. 7 (b)) is calculated as dist.
In this embodiment, the process of determining the target drill pipe and the key points of the target drill pipe is as follows: (1) during drilling: when a target drill rod is detected, directly adding the corresponding center point coordinate, the current video frame number and the current time stamp of the drill rod into an array, defining the array outside a for cycle of an integral data set, and performing mask visualization on the drill rod which is screened to meet the condition; and generating a shade on the target drill rod, and determining the key point of the target drill rod. (2) during the drill withdrawal: when a target drill rod is detected, firstly judging whether coordinate values of key points of the target drill rod on the target drill rod are in a whole body frame of a drilling machine, if so, adding a corresponding center point coordinate of the drill rod, a current video frame number and a current time stamp into an array, wherein the array is defined outside a for cycle of a whole data set, and performing mask visualization on the drill rod which is screened out to meet the condition; and generating a shade on the target drill rod, and determining the key points of the target drill rod.
Step S14: and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
In this embodiment, if the state is drill-in, calculating a drill-in distance between the target drill rod key point and the impact power head center point according to a preset drill-in distance calculation method, drawing a drill-in peak value change graph according to the drill-in distance, and counting drill rods by using the drill-in peak value change graph; if the state is drill withdrawal, calculating the drill withdrawal distance between the key point of the target drill rod and the central point of the impact power head according to a preset drill withdrawal distance calculation method, drawing a drill withdrawal peak value change graph according to the drill withdrawal distance, and counting the drill rods by using the drill withdrawal peak value change graph.
Specifically, counting is performed by the change in distance between the target drill rod key point and the impact power head center point. As shown in fig. 8, the abscissa is the number of video frames (25 frames per second) and the ordinate is the distance dist between the target drill pipe key point and the impact head center point as a function of the number of frames. The distance dist is the difference between the key point of the target drill rod and the central point of the impact power head. Drill rod counting is aided by the peak change in distance dist. The flow is as follows: a threshold value threshold is initialized (the value of threshold is set according to the actual situation). (1) looking for all peaks from the ordinate: when the value of the distance dist between the target drill pipe key point and the impact head center point (i.e., a certain point in fig. 8) is higher than the vertical elevation of the left or right point above threshold, the point is noted as peak. (2) determining the effective peak from the abscissa consideration: the found peaks are screened. Defining two variables peak1 and peak2, respectively recording the positions of the previous wave crest and the current wave crest, defining a variable distance, and calculating the distance between the two wave crests, wherein the position of the previous wave crest is added into the screened list only when the distance between the two wave crests is larger than the distance (the value of the distance is set according to the actual situation), so that the drill rod counting is completed once; otherwise, when the distance between the two peaks is smaller than distance, the counting is not performed, and the peak is discarded. (3) continuous counting: and (3) circularly performing the steps (1) and (2) in the array length formed by all peak values, and continuously counting drill rods until stopping drilling. The calculation ideas are adopted by the drill-in count and the drill-out count, and the difference is that two key points (the key point of the target drill rod and the central point of the impact power head are selected differently (as shown in fig. 5 and 7).
In the embodiment, a historical underground drilling video is obtained, the historical underground drilling video is processed and marked to obtain marked drilling data, and a rotating target detection model for a self-adaptive scene is constructed based on a rotating target detection algorithm and by using the marked drilling data; acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result; based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined; and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram. According to the method, the rotating target detection model for the self-adaptive scene is constructed based on the rotating target detection algorithm and by utilizing the drilling data after marking, the current drilling video is identified by utilizing the rotating target detection model, the target drill rod is screened out based on the drill rod counting reasoning algorithm, the peak value change is accurately obtained through calculating the distance change between the key point of the target drill rod and the central point of the impact power head, and the drill rod counting is carried out.
Referring to fig. 9, the embodiment of the invention discloses a coal mine drill rod counting method in a self-adaptive scene, which specifically comprises the following steps:
step S21: acquiring a historical underground drilling video, dividing the historical underground drilling video into pictures, cleaning each picture to obtain the cleaned picture, and marking the cleaned picture by using a preset rotating target detection marking tool to obtain drilling data after marking.
Step S22: training, testing and verifying a preset initial rotation target detection model based on a rotation target detection algorithm and by using the marked drilling data, so as to obtain the rotation target detection model for the self-adaptive scene; the rotation target detection algorithm comprises Yolov5_OBB, yolov7_OBB, yolov8_ OBB, GGHL, PP-Yolove-R, R3Det and MMRotate algorithm.
In the embodiment, a roLabelImg rotating target detection labeling tool is used for carrying out image processing; furthermore, rotation target detection algorithms include, but are not limited to, yolkv5_obb, yolkv7_obb, yolkv8_ OBB, GGHL, PP-yolke-R, R3Det, and mmrotation.
Step S23: and acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result.
Step S24: and screening the drill rods according to the identification result based on a drill rod counting reasoning algorithm and according to a preset drill rod selection principle to obtain a target drill rod, and determining a target drill rod key point and an impact power head center point.
Step S25: and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
The specific flow of the method is shown in fig. 10, (1) a current drilling video of a coal mine underground is obtained; (2) Identifying the current drilling video by using the trained rotating target detection model to obtain an identification result with a rotating boundary frame; (3) And screening the identification result according to a preset drill rod selection principle based on a drill rod counting reasoning algorithm to obtain a target drill rod, determining a target drill rod key point and an impact power head center point, calculating the distance between the target drill rod key point and the impact power head center point, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
The construction process of the rotating target detection model is as follows: the method comprises the steps of (1) obtaining historical underground drilling videos; (2) Processing the historical underground drilling video, and marking by using a roLabelImg rotary target detection marking tool pair to obtain drilling data after marking; (3) And constructing a rotating target detection model for the self-adaptive scene based on the rotating target detection algorithm and by using the drilling data after marking.
In the embodiment, a historical underground drilling video is obtained, the historical underground drilling video is processed and marked to obtain marked drilling data, and a rotating target detection model for a self-adaptive scene is constructed based on a rotating target detection algorithm and by using the marked drilling data; acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result; based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined; and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram. According to the method, the rotating target detection model for the self-adaptive scene is constructed based on the rotating target detection algorithm and by utilizing the drilling data after marking, the current drilling video is identified by utilizing the rotating target detection model, the target drill rod is screened out based on the drill rod counting reasoning algorithm, the peak value change is accurately obtained through calculating the distance change between the key point of the target drill rod and the central point of the impact power head, and the drill rod counting is carried out.
Referring to fig. 11, the embodiment of the invention discloses a coal mine drill rod counting device with a self-adaptive scene, which specifically comprises the following steps:
the model construction module 11 is used for acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data;
the identifying module 12 is configured to obtain a current drilling video, and identify the current drilling video by using the rotating target detection model to obtain an identification result;
the drill rod screening module 13 is used for screening the drill rods according to the identification result based on a drill rod counting reasoning algorithm and a preset drill rod selection principle so as to obtain a target drill rod and determine a target drill rod key point and an impact power head center point;
and the drill rod counting module 14 is used for calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
In the embodiment, a historical underground drilling video is obtained, the historical underground drilling video is processed and marked to obtain marked drilling data, and a rotating target detection model for a self-adaptive scene is constructed based on a rotating target detection algorithm and by using the marked drilling data; acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result; based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined; and calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram. According to the method, the rotating target detection model for the self-adaptive scene is constructed based on the rotating target detection algorithm and by utilizing the drilling data after marking, the current drilling video is identified by utilizing the rotating target detection model, the target drill rod is screened out based on the drill rod counting reasoning algorithm, the peak value change is accurately obtained through calculating the distance change between the key point of the target drill rod and the central point of the impact power head, and the drill rod counting is carried out.
In some specific embodiments, the model building module 11 may specifically include:
the cleaning processing module is used for cutting the historical underground drilling video into pictures and cleaning the pictures to obtain the pictures after the cleaning processing;
the marking module is used for marking the picture after the cleaning treatment by using a preset rotating target detection marking tool so as to obtain marking drilling data after marking.
In some specific embodiments, the model building module 11 may specifically include:
the training test verification module is used for respectively training, testing and verifying a preset initial rotating target detection model based on a rotating target detection algorithm and by utilizing the marked drilling data so as to obtain the rotating target detection model for the self-adaptive scene; the rotation target detection algorithm comprises Yolov5_OBB, yolov7_OBB, yolov8_ OBB, GGHL, PP-Yolove-R, R3Det and MMRotate algorithm.
In some specific embodiments, the identification module 12 may specifically include:
the identification module is used for identifying the current drilling video by utilizing the rotating target detection model so as to obtain the identification result with the rotating boundary box; the identification result comprises a drilling machine whole, a percussion power head, a drilling machine drill rod and corresponding center point coordinates, short side lengths, long side lengths and rotation angles.
In some embodiments, the drill rod screening module 13 may specifically include:
the principle construction module is used for constructing the drill rod selection principle; the drill rod selection principle comprises a rotation boundary frame selection principle, a shielding exclusion selection principle and a focused drill rod selection principle;
and the drill rod screening module is used for screening the drill rods with the identification result of the rotating boundary frame based on a drill rod counting reasoning algorithm according to the rotating boundary frame selection principle, the shielding exclusion selection principle and the focusing drill rod selection principle so as to obtain a target drill rod.
In some embodiments, the drill rod screening module 13 may specifically include:
the state determining module is used for determining the current state of the target drill rod; the state comprises drill feeding and drill returning;
and the key point and center point determining module is used for determining corresponding target drill rod key points and impact power head center points based on the states.
In some embodiments, the drill rod counting module 14 may specifically include:
the drill rod feeding counting module is used for calculating the drill feeding distance between the target drill rod key point and the central point of the impact power head according to a preset drill feeding distance calculation method if the state is drill feeding, drawing a drill feeding peak value change graph according to the drill feeding distance, and counting drill rods by using the drill feeding peak value change graph;
And the drill rod returning counting module is used for calculating the drill returning distance between the target drill rod key point and the central point of the impact power head according to a preset drill returning distance calculation method if the state is drill returning, drawing a drill returning peak value change graph according to the drill returning distance, and counting the drill rods by using the drill returning peak value change graph.
Fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program that is loaded and executed by the processor 21 to implement relevant steps in the adaptive scene coal mine drill rod counting method performed by the electronic device disclosed in any of the foregoing embodiments.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon include an operating system 221, a computer program 222, and data 223, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the data 223 in the memory 22 by the processor 21, which may be Windows, unix, linux or the like. The computer program 222 may further comprise a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the coal mine drill pipe counting method of the adaptive scenario disclosed by any of the foregoing embodiments as being performed by the electronic device 20. The data 223 may include, in addition to the data received by the adaptive scene coal mine drill pipe counting device and transmitted by the external device, the data collected by the self input/output interface 25, and so on.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Further, the embodiment of the application also discloses a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is loaded and executed by a processor, the method steps of the coal mine drill rod counting method in the self-adaptive scene disclosed in any embodiment are realized.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The method, the device, the equipment and the storage medium for counting the coal mine drill pipes in the self-adaptive scene provided by the invention are described in detail, and specific examples are applied to the description of the principle and the implementation mode of the invention, and the description of the examples is only used for helping to understand the method and the core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims (10)

1. The coal mine drill rod counting method of the self-adaptive scene is characterized by comprising the following steps of:
acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for a self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data;
acquiring a current drilling video, and identifying the current drilling video by using the rotating target detection model to obtain an identification result;
based on a drill rod counting reasoning algorithm, drill rod screening is carried out on the identification result according to a preset drill rod selection principle, so that a target drill rod is obtained, and a target drill rod key point and an impact power head center point are determined;
And calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
2. The adaptive scene coal mine drill pipe counting method according to claim 1, wherein the processing and marking the historical downhole drilling video to obtain marked drilling data comprises:
dividing the historical underground drilling video into pictures, and cleaning each picture to obtain the cleaned picture;
and marking the picture after the cleaning treatment by using a preset rotating target detection marking tool so as to obtain marking drilling data after marking.
3. The adaptive scene coal mine drill pipe counting method according to claim 1, wherein the constructing a rotational target detection model for the adaptive scene based on the rotational target detection algorithm and using the post-annotation drill data comprises:
training, testing and verifying a preset initial rotation target detection model based on a rotation target detection algorithm and by using the marked drilling data, so as to obtain the rotation target detection model for the self-adaptive scene; the rotation target detection algorithm comprises a YOLOv5 OBB, a YOLOv7 OBB, a YOLOv8 OBB, GGHL, PP-YOLOE-R, R Det and an MMRotate algorithm.
4. The adaptive scene coal mine drill rod counting method according to claim 1, wherein the identifying the current drilling video by using the rotation target detection model to obtain an identification result comprises:
identifying the current drilling video by using the rotating target detection model to obtain the identification result with the rotating boundary box; the identification result comprises a drilling machine whole, a percussion power head, a drilling machine drill rod and corresponding center point coordinates, short side lengths, long side lengths and rotation angles.
5. The adaptive scene coal mine drill rod counting method according to claim 4, wherein the drill rod screening is performed on the identification result according to a preset drill rod selection principle based on a drill rod counting reasoning algorithm to obtain a target drill rod, and the method comprises the following steps:
constructing the drill rod selection principle; the drill rod selection principle comprises a rotation boundary frame selection principle, a shielding exclusion selection principle and a focused drill rod selection principle;
and based on a drill rod counting reasoning algorithm, and according to the rotating boundary box selection principle, the shielding exclusion selection principle and the focused drill rod selection principle, drill rod screening is carried out on the identification result with the rotating boundary box so as to obtain a target drill rod.
6. The adaptive scene coal mine drill pipe counting method of any one of claims 1-5, wherein the determining the target drill pipe key point and the impact power head center point comprises:
determining the current state of the target drill rod; the state comprises drill feeding and drill returning;
and determining corresponding target drill rod key points and impact power head center points based on the states.
7. The adaptive scene coal mine drill rod counting method according to claim 6, wherein the calculating in real time the distance between the target drill rod key point and the impact power head center point, drawing a peak change map according to the distance, and counting drill rods by using the peak change map comprises:
if the state is drilling, calculating the drilling distance between the key point of the target drill rod and the central point of the impact power head according to a preset drilling distance calculation method, drawing a drilling peak value change graph according to the drilling distance, and counting the drill rods by using the drilling peak value change graph;
if the state is drill withdrawal, calculating the drill withdrawal distance between the key point of the target drill rod and the central point of the impact power head according to a preset drill withdrawal distance calculation method, drawing a drill withdrawal peak value change graph according to the drill withdrawal distance, and counting the drill rods by using the drill withdrawal peak value change graph.
8. The utility model provides a colliery drilling rod counting assembly of self-adaptation scene which characterized in that includes:
the model construction module is used for acquiring a historical underground drilling video, processing and marking the historical underground drilling video to obtain marked drilling data, and constructing a rotating target detection model for the self-adaptive scene based on a rotating target detection algorithm and by using the marked drilling data;
the identification module is used for acquiring a current drilling video, and identifying the current drilling video by utilizing the rotating target detection model so as to obtain an identification result;
the drill rod screening module is used for screening the drill rods according to the identification result based on a drill rod counting reasoning algorithm and a preset drill rod selection principle so as to obtain a target drill rod and determine a target drill rod key point and an impact power head center point;
and the drill rod counting module is used for calculating the distance between the key point of the target drill rod and the central point of the impact power head in real time, drawing a peak change diagram according to the distance, and counting the drill rods by using the peak change diagram.
9. An electronic device, comprising:
a memory for storing a computer program;
A processor for executing the computer program to implement the adaptive scene coal mine drill rod counting method of any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the coal mine drill rod counting method of the adaptive scenario of any one of claims 1 to 7.
CN202410030225.5A 2024-01-09 2024-01-09 Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene Active CN117830908B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410030225.5A CN117830908B (en) 2024-01-09 2024-01-09 Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410030225.5A CN117830908B (en) 2024-01-09 2024-01-09 Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene

Publications (2)

Publication Number Publication Date
CN117830908A true CN117830908A (en) 2024-04-05
CN117830908B CN117830908B (en) 2024-10-15

Family

ID=90513328

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410030225.5A Active CN117830908B (en) 2024-01-09 2024-01-09 Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene

Country Status (1)

Country Link
CN (1) CN117830908B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112580609A (en) * 2021-01-26 2021-03-30 南京北路智控科技股份有限公司 Coal mine drill rod counting method
US20210115742A1 (en) * 2019-10-16 2021-04-22 Magnetic Variation Services LLC Drill pipe tally system
CN113111805A (en) * 2021-04-16 2021-07-13 北京科技大学 Method for counting underground drilling number of coal mine based on machine vision
CN114821453A (en) * 2022-06-30 2022-07-29 广州英码信息科技有限公司 Coal mine drill rod counting method based on target detection and computer readable medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210115742A1 (en) * 2019-10-16 2021-04-22 Magnetic Variation Services LLC Drill pipe tally system
CN112580609A (en) * 2021-01-26 2021-03-30 南京北路智控科技股份有限公司 Coal mine drill rod counting method
CN113111805A (en) * 2021-04-16 2021-07-13 北京科技大学 Method for counting underground drilling number of coal mine based on machine vision
CN114821453A (en) * 2022-06-30 2022-07-29 广州英码信息科技有限公司 Coal mine drill rod counting method based on target detection and computer readable medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
XIAOJUN WU ET AL: "Drill Pipe Counting Method in Coal Mine Based on Improved Object Tracking", 《2023 8TH INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTING AND SIGNAL PROCESSING (ICSP)》, 31 December 2023 (2023-12-31), pages 1181 - 1184 *
张栋等: "基于改进 MobileNetV2 的钻杆计数方法", 《工矿自动化》, vol. 48, no. 10, 31 October 2022 (2022-10-31) *

Also Published As

Publication number Publication date
CN117830908B (en) 2024-10-15

Similar Documents

Publication Publication Date Title
CN109993780B (en) Three-dimensional high-precision map generation method and device
CN107728615B (en) self-adaptive region division method and system
US20220198688A1 (en) Laser coarse registration method, device, mobile terminal and storage medium
CN110866497B (en) Robot positioning and mapping method and device based on dotted line feature fusion
CN111728535B (en) Method and device for generating cleaning path, electronic equipment and storage medium
CN113435456A (en) Rock slice component identification method and device based on machine learning and medium
CN104822009A (en) Method for identifying video scene transformation and device thereof
CN115546221B (en) Reinforcing steel bar counting method, device, equipment and storage medium
CN104715250A (en) Cross laser detection method and device
CN113925390A (en) Cross-regional channel identification method based on map image, robot and chip
CN117830908B (en) Coal mine drill rod counting method, device, equipment and medium of self-adaptive scene
CN116958889A (en) Semi-supervised small sample target detection method based on pseudo tag
CN113989814B (en) Image generation method, device, computer equipment and storage medium
Yang Enhancing action recognition of construction workers using data-driven scene parsing
CN114387658A (en) Image target attribute detection method, device, equipment and storage medium
CN113741480A (en) Obstacle avoidance method based on combination of dynamic obstacle extraction and cost map
CN117711063A (en) Target behavior detection method, device, equipment and medium
CN108133116A (en) CAD floor plan wall thickness automatic identifying methods and system
CN116958529A (en) Robot remote control method, system and medium
CN102053837B (en) Collision detection and avoidance method and device for space entity element marking
CN117351409A (en) Intelligent concrete dam face operation risk identification method
CN109855634B (en) Image processing method and device for grid map
CN116576861A (en) Visual language navigation method, system, device and storage medium
CN113756795B (en) Oil gas dessert identification method, device, equipment and storage medium
US6868181B1 (en) Method and device for determining a similarity of measure between a first structure and at least one predetermined second structure

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant