CN118155359A - Electric shovel operation safety warning method and system based on double-channel self-learning - Google Patents
Electric shovel operation safety warning method and system based on double-channel self-learning Download PDFInfo
- Publication number
- CN118155359A CN118155359A CN202410573824.1A CN202410573824A CN118155359A CN 118155359 A CN118155359 A CN 118155359A CN 202410573824 A CN202410573824 A CN 202410573824A CN 118155359 A CN118155359 A CN 118155359A
- Authority
- CN
- China
- Prior art keywords
- obstacle
- reflected wave
- image
- electric shovel
- alarm signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000001514 detection method Methods 0.000 claims abstract description 81
- 238000000605 extraction Methods 0.000 claims abstract description 16
- 238000012549 training Methods 0.000 claims description 44
- 230000006870 function Effects 0.000 claims description 13
- 238000012544 monitoring process Methods 0.000 abstract description 5
- 230000002093 peripheral effect Effects 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000009545 invasion Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000033001 locomotion Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Component Parts Of Construction Machinery (AREA)
Abstract
The invention discloses an electric shovel operation safety warning method and system based on double-channel self-learning, and relates to the technical field of excavators, wherein the method comprises the following steps: acquiring millimeter wave radar signals and area images of an external range of three sides of the electric shovel; according to the full-coverage characteristic reflecting surface and the effective reflected wave sequence corresponding to the obstacle, carrying out cross-section reflected wave characteristic extraction and identification on the millimeter wave radar signal, judging whether the obstacle enters, and if so, generating a secondary radar identification alarm signal; inputting the regional image into an obstacle detection model to perform obstacle detection and identification to obtain a corresponding obstacle detection and identification result, and generating a secondary image identification alarm signal if an obstacle exists in the image; when the identification alarm signal is generated, a first-level alarm signal is generated and the electric shovel is controlled to stop acting. The invention can realize the safety warning monitoring of the peripheral area of the electric shovel, effectively identify the intrusion operation area of vehicles and personnel, and reduce false alarm and false alarm conditions.
Description
Technical Field
The invention relates to the technical field of excavators, in particular to an electric shovel operation safety warning method and system based on double-channel self-learning.
Background
The electric shovel is a mechanical electric excavator which transmits power by using transmission parts such as gears, chains, steel rope pulley blocks and the like. When the electric shovel is used, an operator performs excavation control in a manual operation mode, so that safety warning of personnel, vehicles and the like around the electric shovel is performed at present, comprehensive judgment is mainly performed by means of operation experience of intercom equipment and the electric shovel operator, and a large potential safety hazard exists. In order to solve the above-mentioned problems, there is a prior art that an ultrasonic detection device is mounted on a rotary platform of an electric shovel, and an obstacle is detected by the ultrasonic detection device to realize safety warning monitoring around the electric shovel.
However, because the earth surface of the working area of the electric shovel is generally uneven, and the electric shovel can move back and forth along with the digging action, the ultrasonic wave is only used for detecting the invasion obstacle around the electric shovel, and the error detection is easy to be generated due to frequent relative movement of the surrounding environment to the electric shovel, so that the safety production of the electric shovel is seriously affected.
Disclosure of Invention
In order to solve part or all of the technical problems in the prior art, the invention provides an electric shovel operation safety warning method and system based on double-channel self-learning.
The technical scheme of the invention is as follows:
in a first aspect, there is provided an electric shovel operation safety guard method based on dual-channel self-learning, the method comprising:
Acquiring millimeter wave radar signals and area images of an external range outside three sides except a working surface;
According to a fully covered characteristic reflecting surface corresponding to a pre-constructed obstacle and a corresponding effective reflected wave sequence thereof, carrying out cross section reflected wave characteristic extraction and identification on an obtained millimeter wave radar signal, judging whether an obstacle enters, and if so, generating a secondary radar identification alarm signal, wherein the obstacle comprises an operator and an operation vehicle;
Inputting the acquired region image into a pre-trained obstacle detection model to detect and identify the obstacle, obtaining an obstacle detection and identification result corresponding to the region image, and generating a secondary image identification alarm signal if the obstacle exists in the region image;
when the secondary radar identification alarm signal and/or the secondary image identification alarm signal are generated, a primary alarm signal is generated and the electric shovel is controlled to stop acting.
In some possible implementations, the operator's corresponding full-coverage characteristic reflective surface and its corresponding effective reflected wave sequence are constructed in the following manner:
Extracting the outline dimension characteristics and the posture characteristics of an operator;
Based on the outline dimension characteristics and the posture characteristics of operators, constructing a plurality of characteristic reflecting surfaces which correspond to the operators and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
In some possible implementations, the corresponding full-coverage characteristic reflective surface of the work vehicle and its corresponding effective reflected wave sequence are constructed in the following manner:
extracting length, width and height dimension characteristics and vehicle appearance characteristics of the working vehicle;
based on the length, width and height characteristics of the working vehicle and the appearance characteristics of the vehicle, constructing a plurality of characteristic reflecting surfaces which correspond to the working vehicle and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
In some possible implementation manners, the cross-section reflected wave feature extraction and identification are performed on the obtained millimeter wave radar signals by using the following modes to judge whether an obstacle enters:
extracting a corresponding characteristic reflecting surface according to the obtained millimeter wave radar signal, and judging whether the characteristic reflecting surface forms a real-time reflected wave sequence or not;
if the real-time reflected wave sequence is formed, comparing the real-time reflected wave sequence with all the effective reflected wave sequences, if the effective reflected wave sequence is matched with the real-time reflected wave sequence, determining an obstacle corresponding to the matched effective reflected wave sequence, extracting a corresponding characteristic reflecting surface according to the acquired next millimeter wave radar signal, and judging whether the characteristic reflecting surface forms the real-time reflected wave sequence or not;
If the real-time reflected wave sequence is formed, comparing the real-time reflected wave sequence with all the effective reflected wave sequences, and if the effective reflected wave sequence corresponding to the obstacle determined in the previous step can be matched with the real-time reflected wave sequence, judging that the obstacle enters a set range area, wherein the entered obstacle is the obstacle determined in the previous step;
And if the coincidence ratio of the effective reflected wave sequence and the real-time reflected wave sequence is more than 85%, the effective reflected wave sequence is matched with the real-time reflected wave sequence.
In some possible implementations, the obstacle detection model employs the YOLO series algorithm or FASTER RCNN algorithm.
In some possible implementations, the obstacle detection model is trained by:
acquiring an image training sample set, wherein the image training sample comprises: a sample image including an obstacle, an obstacle type and a position corresponding to the sample image;
and training an obstacle detection model by taking a sample image in the image training sample as input and taking the type and the position of an obstacle corresponding to the sample image as output.
In some possible implementations, taking a sample image in the image training sample as an input, taking an obstacle type and a position corresponding to the sample image as an output, training an obstacle detection model, including:
Step S321, sequentially inputting sample images in each image training sample in the image training sample set into an obstacle detection model to obtain the predicted obstacle type and position output by the obstacle detection model;
Step S322, calculating a preset target detection loss function according to the type and the position of the obstacle corresponding to the sample image in each image training sample and the type and the position of the predicted obstacle corresponding to the sample image and output by the obstacle detection model;
step S323, judging whether a preset training stopping condition is met, if yes, taking the current obstacle detection model as an obstacle detection model for completing training, if not, updating parameters of the obstacle detection model by using a preset target detection loss function, and returning to step S321.
In some possible implementations, the method further includes:
saving the acquired region image with the obstacle;
And when the stored area images reach the set quantity, updating and learning the obstacle detection model by using the stored area images.
In a second aspect, there is also provided an electric shovel operation safety guard system based on dual-channel self-learning, the system comprising:
an information acquisition unit configured to acquire millimeter wave radar signals and area images of an external range of three sides other than the electric shoveling working face;
The reflected wave feature extraction and identification unit is configured to extract and identify the cross-section reflected wave features of the obtained millimeter wave radar signal according to the fully covered feature reflecting surface corresponding to the pre-constructed obstacle and the corresponding effective reflected wave sequence thereof, judge whether the obstacle enters, generate a secondary radar identification alarm signal and send the secondary radar identification alarm signal to the control unit if the obstacle enters;
The obstacle detection and recognition unit is configured to input the acquired area image into a pre-trained obstacle detection model to detect and recognize the obstacle, obtain an obstacle detection and recognition result corresponding to the area image, generate a secondary image recognition alarm signal and send the secondary image recognition alarm signal to the control unit if the obstacle exists in the area image;
The control unit is connected with the control system of the electric shovel and is configured to generate a first-level alarm signal and control the electric shovel to stop acting when receiving the second-level radar identification alarm signal and/or the second-level image identification alarm signal.
In some possible implementations, the information acquisition unit includes:
The millimeter wave radar comprises six millimeter wave radars, wherein two millimeter wave radars are respectively arranged on each of three side surfaces of the electric shovel, two millimeter wave radars in each side surface are respectively positioned on two sides of the side surface, and the six millimeter wave radars are positioned on the same height;
The industrial camera comprises three sides, and one industrial camera is respectively installed in the middle of each of the three sides of the electric shovel.
The technical scheme of the invention has the main advantages that:
According to the electric shovel operation safety warning method and system based on the double-channel self-learning, millimeter wave radar signals and area images of the outside of the three sides except the working face of the electric shovel are obtained in the operation process of the electric shovel, cross section reflected wave feature extraction and identification are carried out on the millimeter wave radar signals to judge whether an obstacle enters, detection and identification are carried out on the area images to judge whether the obstacle enters, and warning is carried out and the electric shovel is controlled to stop when the obstacle enters is judged, so that safety warning monitoring of the peripheral area of the electric shovel can be achieved, vehicles and personnel invading the operation area can be effectively identified, collision between the vehicles and the personnel and the electric shovel is avoided, operation safety of the electric shovel is guaranteed, and misinformation conditions can be remarkably reduced.
Drawings
The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and without limitation to the invention. In the drawings:
FIG. 1 is a flow chart of a method for guard of electric shovel operation safety based on dual-channel self-learning according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an electric shovel operation safety warning system based on two-channel self-learning according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to specific embodiments of the present invention and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The following describes in detail the technical scheme provided by the embodiment of the invention with reference to the accompanying drawings.
Referring to fig. 1, in a first aspect, an embodiment of the present invention provides a dual-channel self-learning-based electric shovel operation safety alert method, which includes the following steps:
And S1, acquiring millimeter wave radar signals and area images of which the outer ranges are arranged outside three sides except the electric shoveling working surface.
Specifically, during the operation of the electric shovel, millimeter wave radar signals and area images of the external ranges of the three sides of the electric shovel outside the working surface are acquired in real time at a given frequency. The setting range is specifically set according to the actual situation.
Because the working face of the electric shovel is positioned in front of the cab of the electric shovel, operators in the cab can directly observe and know the situation in front of the working face, and therefore, only the situation that the electric shovel is except for three sides of the working face is needed to be obtained.
Further, the acquiring frequencies of the millimeter wave radar signal and the area image are specifically set according to the actual situation, and the acquiring frequencies of the millimeter wave radar signal and the area image may be the same or different. For example, the two times are set to 2-5 s/time.
Further, in an embodiment of the present invention, the millimeter wave radar signal is acquired by a millimeter wave radar mounted on the side of the electric shovel, and the area image is acquired by an industrial camera mounted on the side of the electric shovel.
Specifically, the millimeter wave radar for acquiring millimeter wave radar signals comprises six, two millimeter wave radars are respectively installed on each of three sides of the electric shovel, the two millimeter wave radars in each side are respectively located on two sides of the side, and the six millimeter wave radars are located on the same height.
By arranging the six millimeter wave radars as above, the full coverage of the millimeter wave radars to the peripheral area of the side face of the electric shovel except the working face can be realized.
Specifically, the industrial cameras for acquiring the area image include three, one industrial camera is respectively mounted in the middle of each of the three sides of the electric shovel, and the mounting height of the industrial camera is the same as that of the millimeter wave radar.
By providing the three industrial cameras as described above, the industrial cameras can fully cover the peripheral area of the side face of the electric shovel other than the working face.
And S2, performing cross-section reflected wave feature extraction and identification on the obtained millimeter wave radar signal according to the fully covered feature reflecting surface corresponding to the pre-constructed obstacle and the corresponding effective reflected wave sequence, judging whether the obstacle enters, and if so, generating a secondary radar identification alarm signal.
In one embodiment of the invention, the obstacle comprises an operator and a work vehicle. The operation personnel determine according to the operation scene and the operation environment of the actual electric shovel, and the operation vehicle comprises a mine truck and other vehicles possibly entering the operation area of the electric shovel except the mine truck, wherein the mine truck refers to a mining transport vehicle.
Specifically, before performing the operation of the electric shovel, each operator who may enter the operation area of the electric shovel and each type of operation vehicle which may enter the operation area of the electric shovel are determined. And constructing a corresponding full-coverage characteristic reflecting surface and a corresponding effective reflected wave sequence of the full-coverage characteristic reflecting surface aiming at each operator and each type of working vehicle.
In an embodiment of the present invention, for an operator, a full coverage characteristic reflecting surface corresponding to the operator and an effective reflected wave sequence corresponding to the full coverage characteristic reflecting surface are constructed in the following manner:
Extracting the outline dimension characteristics and the posture characteristics of an operator;
Based on the outline dimension characteristics and the posture characteristics of operators, constructing a plurality of characteristic reflecting surfaces which correspond to the operators and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
For a work vehicle, the corresponding full-coverage characteristic reflective surface of the work vehicle and its corresponding effective reflected wave sequence are constructed in the following manner:
extracting length, width and height dimension characteristics and vehicle appearance characteristics of the working vehicle;
based on the length, width and height characteristics of the working vehicle and the appearance characteristics of the vehicle, constructing a plurality of characteristic reflecting surfaces which correspond to the working vehicle and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
Further, since the operator or the working vehicle enters the millimeter wave radar monitoring area in any posture, the operator or the working vehicle faces the millimeter wave radar with the approximate characteristic reflecting surface, in one embodiment of the invention, the cross-section reflected wave characteristic extraction and identification are performed on the obtained millimeter wave radar signal by using the following method to judge whether an obstacle enters:
step S201, extracting a corresponding characteristic reflecting surface according to the obtained millimeter wave radar signal, and judging whether the characteristic reflecting surface forms a real-time reflected wave sequence or not;
Step S202, if a real-time reflected wave sequence is formed, comparing the real-time reflected wave sequence with all effective reflected wave sequences, if the effective reflected wave sequence is matched with the real-time reflected wave sequence, determining an obstacle corresponding to the matched effective reflected wave sequence, extracting a corresponding characteristic reflecting surface according to the acquired next millimeter wave radar signal, and judging whether the characteristic reflecting surface forms the real-time reflected wave sequence or not;
In step S203, if a real-time reflected wave sequence is formed, the real-time reflected wave sequence is compared with all the effective reflected wave sequences, and if an effective reflected wave sequence corresponding to the obstacle determined in the previous step can be matched with the real-time reflected wave sequence, it is determined that an obstacle enters the set range area, and the entered obstacle is the obstacle determined in the previous step.
In an embodiment of the present invention, the processing described in steps S201 to S203 is performed for each acquired millimeter wave radar signal. As can be seen from the processing in steps S201 to S203, only when the effective reflected wave sequence belonging to the same obstacle can be matched with the real-time reflected wave sequence corresponding to the millimeter wave radar signal acquired twice consecutively, it is determined that the obstacle enters the set range region. Therefore, the probability of erroneous judgment and erroneous judgment can be reduced, the safe operation of the electric shovel is ensured, and the influence on the normal operation of the electric shovel is avoided.
In an embodiment of the present invention, a determination manner for matching an effective reflected wave sequence with a real-time reflected wave sequence is set based on a full-coverage characteristic reflecting surface corresponding to an obstacle and a construction manner of the corresponding effective reflected wave sequence.
Specifically, for a plurality of characteristic reflection surfaces covering 360 ° full viewing angle corresponding to an obstacle constructed at intervals of 5 °, a determination mode of matching an effective reflection wave sequence with a real-time reflection wave sequence is set as follows: if the coincidence ratio of the effective reflected wave sequence and the real-time reflected wave sequence is more than 85%, the effective reflected wave sequence is matched with the real-time reflected wave sequence.
In an embodiment of the present invention, a plurality of millimeter wave radar signals acquired at the same time are processed in parallel.
And step S3, inputting the acquired region image into a pre-trained obstacle detection model to detect and identify the obstacle, obtaining an obstacle detection and identification result corresponding to the region image, and generating a secondary image identification alarm signal if the obstacle exists in the region image.
In an embodiment of the present invention, the execution sequence of step S2 and step S3 is not related, and may be executed synchronously, or executed in parallel, or step S2 may be executed first, or step S3 may be executed first, where the execution time of both steps depends on the acquisition time and frequency of the millimeter wave radar signal and the area image, step S2 is executed after the millimeter wave radar signal is acquired, and step S3 is executed after the area image is acquired.
In an embodiment of the present invention, the obstacle detection model adopts YOLO series algorithm or FASTER RCNN algorithm.
Taking FASTER RCNN algorithm as an example, the FASTER RCNN algorithm firstly extracts a feature map through a feature extraction layer, for example, a classification model is used as a basic network, and a convolution operation of the basic network is used to obtain the feature map; then, using RPN (Region Proposal Network, candidate area network) to calculate if a certain area of the original image contains a specific object; if the object is contained, carrying out feature extraction by using a ROI (region of interest) pooling layer, and then predicting the object category and a boundary box (bounding box) through a classification regression layer; if no object is contained, no classification is made. The base network of the feature extraction layer may be AlexNet, VGG, googleNet, resNet or the like.
Further, in an embodiment of the present invention, the obstacle detection model is trained by:
Step S31, acquiring an image training sample set, where the image training sample includes: a sample image including an obstacle, an obstacle type and a position corresponding to the sample image;
Step S32, taking a sample image in the image training sample as input, taking the type and the position of the obstacle corresponding to the sample image as output, and training an obstacle detection model.
In an embodiment of the present invention, the image training sample may be selected from existing images including the obstacle, or may be captured by using a corresponding industrial camera to obtain the images including the obstacle. The obstacle includes an operator and a work vehicle.
The number of image training samples in the image training sample set is determined according to actual requirements, and in general, the greater the number of image training samples, the higher the accuracy of the trained obstacle detection model, but the higher the required sample acquisition cost and training cost.
Further, in an embodiment of the present invention, a sample image in an image training sample is taken as an input, an obstacle type and a position corresponding to the sample image are taken as an output, and an obstacle detection model is trained, including the steps of:
Step S321, sequentially inputting sample images in each image training sample in the image training sample set into the obstacle detection model to obtain the predicted obstacle type and position output by the obstacle detection model.
In one embodiment of the invention, a sample image in an image training sample is input from an input end of an obstacle detection model, sequentially processed by parameters of each layer in the obstacle detection model, and output from an output end of the obstacle detection model, wherein the information output by the output end of the obstacle detection model is the predicted obstacle type and position corresponding to the sample image.
In an embodiment of the present invention, the obstacle detection model is a model to be trained, parameters of each layer of the model are initialization parameters, and parameters of each layer of the model are continuously updated in the training process of the model.
Step S322, calculating a preset target detection loss function according to the type and position of the obstacle corresponding to the sample image in each image training sample and the type and position of the predicted obstacle output by the obstacle detection model corresponding to the sample image.
In an embodiment of the present invention, the objective detection loss function is specifically set according to the actual situation. For example, a conventional target detection loss function is adopted, specifically expressed as:
；
Wherein, Representing classification loss,/>Indicating a loss of position. The classification Loss and the location Loss may employ conventional Loss functions, for example, the classification Loss employs a cross entropy Loss function, the location Loss employs a Smooth L1 Loss or an IOU Loss, or the like.
Specifically, based on the set target detection loss function, the target detection loss function is calculated from the type and position of the obstacle corresponding to the sample image in each image training sample, and the type and position of the predicted obstacle output by the obstacle detection model corresponding to the sample image.
Step S323, judging whether a preset training stopping condition is met, if yes, taking the current obstacle detection model as an obstacle detection model for completing training, if not, updating parameters of the obstacle detection model by using a preset target detection loss function, and returning to step S321.
In an embodiment of the present invention, the training stop condition is specifically set according to the actual situation, for example, the training iteration number reaches a set iteration algebra; or the target detection loss function is used as an optimization index of the obstacle model, and the training stopping condition is set to be that the optimization index reaches a set threshold.
In an embodiment of the present invention, a random gradient descent method is used to perform training and updating of parameters of the obstacle detection model.
In an embodiment of the present invention, when the obstacle detection model performs obstacle detection and recognition, the corresponding detection confidence threshold is specifically set according to the actual requirement, for example, set to be more than 90%.
Further, the electric shovel operation safety warning method provided by the embodiment of the invention further comprises the following steps:
saving the acquired region image with the obstacle;
And when the stored area images reach the set quantity, updating and learning the obstacle detection model by using the stored area images.
In this way, the detection and recognition accuracy of the obstacle detection model can be improved continuously by continuously learning and updating the model.
Correspondingly, if no obstacle exists in the acquired area image, and the second-level radar identification alarm signal is not generated according to the millimeter wave radar signal acquired at the time closest to the acquisition time of the area image, the corresponding area image is cleared, so that the storage cost of the image is reduced.
In an embodiment of the present invention, a plurality of area images acquired at the same time are processed in parallel.
And S4, when a secondary radar identification alarm signal and/or a secondary image identification alarm signal are generated, generating a primary alarm signal and controlling the electric shovel to stop acting.
Specifically, when at least one of the secondary radar identification alarm signal and the secondary image identification alarm signal is generated, a primary alarm signal is generated and the electric shovel is controlled to stop acting.
Further, in order to improve the operation efficiency of the electric shovel, in an embodiment of the present invention, the method for guard of operation safety of the electric shovel further includes:
If the electric shovel is in a loading state and the obstacle entering the set range is detected to be a mine card, a primary alarm signal is not generated and a secondary alarm signal is released.
Wherein, the second-level alarm signal includes: the secondary radar recognizes the alarm signal and the secondary image recognizes the alarm signal.
Therefore, the influence on normal production of the electric shovel can be reduced while the safety guard production of the electric shovel is realized.
Referring to fig. 2, in a second aspect, an embodiment of the present invention further provides an electric shovel operation safety alert system based on dual-channel self-learning, the system including:
an information acquisition unit 100 configured to acquire millimeter wave radar signals and area images of an external range of three sides other than the electric power shoveling work surface;
the reflected wave feature extraction and identification unit 200 is connected with the information acquisition unit 100 and the control unit 400 respectively, and is configured to perform cross-section reflected wave feature extraction and identification on the obtained millimeter wave radar signal according to the fully covered feature reflecting surface corresponding to the pre-constructed obstacle and the corresponding effective reflected wave sequence thereof, and judge whether the obstacle enters, if yes, generate a secondary radar identification alarm signal and send the secondary radar identification alarm signal to the control unit 400;
The obstacle detection and recognition unit 300 is connected with the information acquisition unit 100 and the control unit 400 respectively, and is configured to input the acquired area image into a pre-trained obstacle detection model to perform obstacle detection and recognition, obtain an obstacle detection and recognition result corresponding to the area image, and generate a secondary image recognition alarm signal and send the secondary image recognition alarm signal to the control unit 400 if an obstacle exists in the area image;
And the control unit 400 is connected with the control system of the electric shovel and is configured to generate a primary alarm signal and control the electric shovel to stop acting when receiving the secondary radar identification alarm signal and/or the secondary image identification alarm signal.
Further, in an embodiment of the present invention, the information acquisition unit 100 includes:
the millimeter wave radar comprises six millimeter wave radars, wherein two millimeter wave radars are respectively arranged on each of three side surfaces of the electric shovel, the two millimeter wave radars in each side surface are respectively positioned on two sides of the side surface, and the six millimeter wave radars are positioned on the same height;
The industrial camera comprises three sides, and one industrial camera is respectively installed in the middle of each of the three sides of the electric shovel.
Specifically, the millimeter wave radar is installed under the two-layer platform of the electric shovel, and the installation height of the millimeter wave radar is the same as that of the industrial camera.
The above units are devices corresponding to the steps of the above method, and the specific working principle and the beneficial effects of each unit can be referred to the above method, which is not described herein.
According to the electric shovel operation safety warning method and system based on the double-channel self-learning, provided by the embodiment of the invention, the millimeter wave radar signals and the area images with the outside of the three sides except the working surface of the electric shovel are obtained in the operation process of the electric shovel, the section reflected wave feature extraction recognition is carried out on the millimeter wave radar signals to judge whether an obstacle enters, the detection recognition is carried out on the area images to judge whether the obstacle enters, and the alarm is carried out and the electric shovel is controlled to stop moving when the obstacle enters is judged, so that the safety warning monitoring of the peripheral area of the electric shovel can be realized, the invasion of vehicles and personnel into the operation area can be effectively identified, the collision between the vehicles and personnel and the electric shovel is avoided, the operation safety of the electric shovel is ensured, and the situations of false alarm and false alarm can be remarkably reduced.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting thereof; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The electric shovel operation safety warning method based on the double-channel self-learning is characterized by comprising the following steps of:
Acquiring millimeter wave radar signals and area images of an external range outside three sides except a working surface;
According to a fully covered characteristic reflecting surface corresponding to a pre-constructed obstacle and a corresponding effective reflected wave sequence thereof, carrying out cross section reflected wave characteristic extraction and identification on an obtained millimeter wave radar signal, judging whether an obstacle enters, and if so, generating a secondary radar identification alarm signal, wherein the obstacle comprises an operator and an operation vehicle;
Inputting the acquired region image into a pre-trained obstacle detection model to detect and identify the obstacle, obtaining an obstacle detection and identification result corresponding to the region image, and generating a secondary image identification alarm signal if the obstacle exists in the region image;
when the secondary radar identification alarm signal and/or the secondary image identification alarm signal are generated, a primary alarm signal is generated and the electric shovel is controlled to stop acting.
2. The electric shovel operation safety guard method based on the double-channel self-learning according to claim 1, wherein the full-coverage characteristic reflecting surface corresponding to an operator and the corresponding effective reflected wave sequence are constructed by the following modes:
Extracting the outline dimension characteristics and the posture characteristics of an operator;
Based on the outline dimension characteristics and the posture characteristics of operators, constructing a plurality of characteristic reflecting surfaces which correspond to the operators and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
3. The electric shovel operation safety guard method based on the double-channel self-learning according to claim 1, wherein the full-coverage characteristic reflecting surface corresponding to the operation vehicle and the effective reflected wave sequence corresponding to the full-coverage characteristic reflecting surface are constructed by the following modes:
extracting length, width and height dimension characteristics and vehicle appearance characteristics of the working vehicle;
based on the length, width and height characteristics of the working vehicle and the appearance characteristics of the vehicle, constructing a plurality of characteristic reflecting surfaces which correspond to the working vehicle and cover 360 degrees of full view angles at intervals of 5 degrees;
For each characteristic reflecting surface, a corresponding effective reflected wave sequence is extracted.
4. The electric shovel operation safety guard method based on the double-channel self-learning according to any one of claims 1 to 3, wherein the cross-section reflected wave feature extraction and identification are performed on the obtained millimeter wave radar signal by using the following modes to judge whether an obstacle enters:
extracting a corresponding characteristic reflecting surface according to the obtained millimeter wave radar signal, and judging whether the characteristic reflecting surface forms a real-time reflected wave sequence or not;
if the real-time reflected wave sequence is formed, comparing the real-time reflected wave sequence with all the effective reflected wave sequences, if the effective reflected wave sequence is matched with the real-time reflected wave sequence, determining an obstacle corresponding to the matched effective reflected wave sequence, extracting a corresponding characteristic reflecting surface according to the acquired next millimeter wave radar signal, and judging whether the characteristic reflecting surface forms the real-time reflected wave sequence or not;
If the real-time reflected wave sequence is formed, comparing the real-time reflected wave sequence with all the effective reflected wave sequences, and if the effective reflected wave sequence corresponding to the obstacle determined in the previous step can be matched with the real-time reflected wave sequence, judging that the obstacle enters a set range area, wherein the entered obstacle is the obstacle determined in the previous step;
And if the coincidence ratio of the effective reflected wave sequence and the real-time reflected wave sequence is more than 85%, the effective reflected wave sequence is matched with the real-time reflected wave sequence.
5. The electric shovel operation safety guard method based on the double-channel self-learning according to claim 1, wherein the obstacle detection model adopts a YOLO series algorithm or FASTER RCNN algorithm.
6. The two-channel self-learning-based electric shovel operation safety guard method according to claim 5, wherein the obstacle detection model is trained by:
acquiring an image training sample set, wherein the image training sample comprises: a sample image including an obstacle, an obstacle type and a position corresponding to the sample image;
and training an obstacle detection model by taking a sample image in the image training sample as input and taking the type and the position of an obstacle corresponding to the sample image as output.
7. The two-channel self-learning-based electric shovel operation safety guard method according to claim 6, wherein the training of the obstacle detection model by taking a sample image in the image training sample as an input and taking an obstacle type and a position corresponding to the sample image as an output comprises:
Step S321, sequentially inputting sample images in each image training sample in the image training sample set into an obstacle detection model to obtain the predicted obstacle type and position output by the obstacle detection model;
Step S322, calculating a preset target detection loss function according to the type and the position of the obstacle corresponding to the sample image in each image training sample and the type and the position of the predicted obstacle corresponding to the sample image and output by the obstacle detection model;
step S323, judging whether a preset training stopping condition is met, if yes, taking the current obstacle detection model as an obstacle detection model for completing training, if not, updating parameters of the obstacle detection model by using a preset target detection loss function, and returning to step S321.
8. The two-channel self-learning based electric shovel operation safety guard method according to any one of claims 5 to 7, further comprising:
saving the acquired region image with the obstacle;
And when the stored area images reach the set quantity, updating and learning the obstacle detection model by using the stored area images.
9. Electric shovel operation safety warning system based on binary channels is from study, characterized in that includes:
an information acquisition unit configured to acquire millimeter wave radar signals and area images of an external range of three sides other than the electric shoveling working face;
The reflected wave feature extraction and identification unit is configured to extract and identify the cross-section reflected wave features of the obtained millimeter wave radar signal according to the fully covered feature reflecting surface corresponding to the pre-constructed obstacle and the corresponding effective reflected wave sequence thereof, judge whether the obstacle enters, generate a secondary radar identification alarm signal and send the secondary radar identification alarm signal to the control unit if the obstacle enters;
The obstacle detection and recognition unit is configured to input the acquired area image into a pre-trained obstacle detection model to detect and recognize the obstacle, obtain an obstacle detection and recognition result corresponding to the area image, generate a secondary image recognition alarm signal and send the secondary image recognition alarm signal to the control unit if the obstacle exists in the area image;
The control unit is connected with the control system of the electric shovel and is configured to generate a first-level alarm signal and control the electric shovel to stop acting when receiving the second-level radar identification alarm signal and/or the second-level image identification alarm signal.
10. The two-channel self-learning-based electric shovel operation safety guard system according to claim 9, wherein the information acquisition unit comprises:
The millimeter wave radar comprises six millimeter wave radars, wherein two millimeter wave radars are respectively arranged on each of three side surfaces of the electric shovel, two millimeter wave radars in each side surface are respectively positioned on two sides of the side surface, and the six millimeter wave radars are positioned on the same height;
The industrial camera comprises three sides, and one industrial camera is respectively installed in the middle of each of the three sides of the electric shovel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410573824.1A CN118155359A (en) | 2024-05-10 | 2024-05-10 | Electric shovel operation safety warning method and system based on double-channel self-learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410573824.1A CN118155359A (en) | 2024-05-10 | 2024-05-10 | Electric shovel operation safety warning method and system based on double-channel self-learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118155359A true CN118155359A (en) | 2024-06-07 |
Family
ID=91301621
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410573824.1A Pending CN118155359A (en) | 2024-05-10 | 2024-05-10 | Electric shovel operation safety warning method and system based on double-channel self-learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118155359A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101189533A (en) * | 2005-05-30 | 2008-05-28 | 罗伯特·博世有限公司 | Method and apparatus for identifying and classifying objects |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| CN111344460A (en) * | 2017-12-04 | 2020-06-26 | 住友重机械工业株式会社 | Periphery monitoring device, information processing terminal, information processing device, and information processing program |
| US20200369290A1 (en) * | 2019-05-21 | 2020-11-26 | Deere & Company | System and method for configuring worksite warning zones |
| CN112470033A (en) * | 2018-07-27 | 2021-03-09 | 松下电器产业株式会社 | Radar data processing device, object discrimination device, radar data processing method, and object discrimination method |
| CN114035198A (en) * | 2020-07-29 | 2022-02-11 | 四川鼎鸿智电装备科技有限公司 | Engineering machine and environment detection system thereof |
| CN114332732A (en) * | 2021-12-28 | 2022-04-12 | 四川西南交大铁路发展股份有限公司 | Railway crisis monitoring method based on radar vision fusion |
| CN115244586A (en) * | 2020-03-03 | 2022-10-25 | 苏州七星天专利运营管理有限责任公司 | Microwave identification method and system |
| CN117315620A (en) * | 2023-09-26 | 2023-12-29 | 华能澜沧江水电股份有限公司 | Construction obstacle sensing method, equipment and medium |
-
2024
- 2024-05-10 CN CN202410573824.1A patent/CN118155359A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101189533A (en) * | 2005-05-30 | 2008-05-28 | 罗伯特·博世有限公司 | Method and apparatus for identifying and classifying objects |
| CN111344460A (en) * | 2017-12-04 | 2020-06-26 | 住友重机械工业株式会社 | Periphery monitoring device, information processing terminal, information processing device, and information processing program |
| CN112470033A (en) * | 2018-07-27 | 2021-03-09 | 松下电器产业株式会社 | Radar data processing device, object discrimination device, radar data processing method, and object discrimination method |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| US20200369290A1 (en) * | 2019-05-21 | 2020-11-26 | Deere & Company | System and method for configuring worksite warning zones |
| CN115244586A (en) * | 2020-03-03 | 2022-10-25 | 苏州七星天专利运营管理有限责任公司 | Microwave identification method and system |
| CN114035198A (en) * | 2020-07-29 | 2022-02-11 | 四川鼎鸿智电装备科技有限公司 | Engineering machine and environment detection system thereof |
| CN114332732A (en) * | 2021-12-28 | 2022-04-12 | 四川西南交大铁路发展股份有限公司 | Railway crisis monitoring method based on radar vision fusion |
| CN117315620A (en) * | 2023-09-26 | 2023-12-29 | 华能澜沧江水电股份有限公司 | Construction obstacle sensing method, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102065975B1 (en) | Safety Management System Using a Lidar for Heavy Machine | |
| US11975951B2 (en) | Engineering machinery and dynamic anti-collision method, device, and system for operation space of the engineering machinery | |
| CN103176185B (en) | Method and system for detecting road barrier | |
| EP0567059B1 (en) | Object recognition system using image processing | |
| CN109634282A (en) | Automatic driving vehicle, method and apparatus | |
| CN109344687B (en) | Vision-based obstacle detection method and device and mobile device | |
| CN111622296B (en) | Excavator safety obstacle avoidance system and method | |
| US10151830B2 (en) | Systems and methods for detecting objects proximate to a machine utilizing a learned process | |
| KR101895835B1 (en) | Ground penetrating radar survey system | |
| AU2007201110A1 (en) | Systems and methods for determining threshold warning distances for collision avoidance | |
| CN113096397A (en) | Traffic jam analysis method based on millimeter wave radar and video detection | |
| JPH1072851A (en) | Invading moving body detection device | |
| CN112799339A (en) | Method for early warning workers and dynamic danger sources based on machine vision technology | |
| WO2017150136A1 (en) | External interference removal device | |
| CN118155359A (en) | Electric shovel operation safety warning method and system based on double-channel self-learning | |
| CN117686650A (en) | Intelligent defect detection system and method suitable for tunnel in operation period | |
| CN115373402A (en) | Loader running control method, device and equipment and storage medium | |
| CN116309552B (en) | Method, device, equipment and medium for evaluating health state of existing line old retaining wall | |
| CN114078326B (en) | Collision detection method, device, visual sensor and storage medium | |
| CN112946663B (en) | Grading automatic collision alarm method for forward-looking obstacle-detection sonar | |
| KR102678598B1 (en) | Apparatus for detecting vehicle using LiDAR | |
| JP7175245B2 (en) | working machine | |
| Seo et al. | Radar Signal Abnormal Point Classification based on Camera-Radar Sensor Fusion | |
| CN113158725B (en) | Comprehensive engineering vehicle construction action judgment method | |
| WO2022190500A1 (en) | Travel assistance apparatus, travel assistance control method, and travel assistance control program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |