CN112541413A - Dangerous behavior detection method and system for forklift driver practical operation examination and coaching - Google Patents
Dangerous behavior detection method and system for forklift driver practical operation examination and coaching Download PDFInfo
- Publication number
- CN112541413A CN112541413A CN202011388071.5A CN202011388071A CN112541413A CN 112541413 A CN112541413 A CN 112541413A CN 202011388071 A CN202011388071 A CN 202011388071A CN 112541413 A CN112541413 A CN 112541413A
- Authority
- CN
- China
- Prior art keywords
- driver
- dangerous
- forklift
- posture
- industrial camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 19
- 230000006399 behavior Effects 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000003384 imaging method Methods 0.000 claims abstract description 6
- 238000003709 image segmentation Methods 0.000 claims abstract description 5
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000005484 gravity Effects 0.000 claims description 2
- 238000013527 convolutional neural network Methods 0.000 description 8
- 238000013135 deep learning Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000009182 swimming Effects 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003137 locomotive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
- B66F17/00—Safety devices, e.g. for limiting or indicating lifting force
- B66F17/003—Safety devices, e.g. for limiting or indicating lifting force for fork-lift trucks
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66F—HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD
- B66F9/00—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes
- B66F9/06—Devices for lifting or lowering bulky or heavy goods for loading or unloading purposes movable, with their loads, on wheels or the like, e.g. fork-lift trucks
- B66F9/075—Constructional features or details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/18—Status alarms
- G08B21/24—Reminder alarms, e.g. anti-loss alarms
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B3/00—Audible signalling systems; Audible personal calling systems
- G08B3/10—Audible signalling systems; Audible personal calling systems using electric transmission; using electromagnetic transmission
Abstract
The invention discloses a dangerous behavior detection method and system for forklift driver practical operation examination and coaching, which comprises the following steps: installing an industrial camera at the front end of the forklift, adjusting the angle and the focal length of the camera until a waist safety belt and an upper half part of a forklift driver can be shot clearly, and recording the adjusted angle of the industrial camera; capturing the position image according to the position of the forklift driver, acquiring the position pixel coordinates of the forklift driver by using a Mask R-CNN target recognition and image segmentation frame, and calculating the position angle of the upper half of the driver; obtaining the proportion of the pixel spacing to the actual spacing according to the imaging structure of the camera system, and calculating a dangerous behavior judgment standard; the industrial camera shoots and monitors the posture of a forklift driver, detects whether a safety belt is fastened or not, utilizes Mask R-CNN to identify the pixel coordinates of the posture of the driver, calculates a safety score and judges whether behaviors are dangerous or not. The intelligent identification system realizes intelligent identification and detection of dangerous behaviors of the driver in the actual operation examination and coaching process of the forklift driver.
Description
Technical Field
The invention relates to the field of forklift driver assessment and coaching, in particular to a dangerous behavior detection method and system for forklift driver practical operation assessment and coaching based on deep learning.
Background
The forklift is widely applied to ports, stations, airports, goods yards, factory workshops, warehouses, circulation centers, distribution centers and the like, carries out loading, unloading and carrying operations of pallet goods in cabins, carriages and containers, and is essential equipment in pallet transportation and container transportation. The practical operation examination and the quality of the coaches of the forklift drivers are concerned with the engineering efficiency and the engineering safety. The method relates to a deep learning method, can detect dangerous behaviors of a forklift driver in actual operation examination and coaching processes, is beneficial to improving the teaching and training quality of the forklift driver in industry, standardizes the driving and operating behavior habits of the forklift driver, reduces the occurrence probability of forklift safety accidents from the source, and powerfully ensures the use safety of the forklift.
In the prior art of the invention, the following comparative patents and documents are provided:
1) a bus driver violation detection system (CN 109376634A) based on a neural network discloses a bus driver violation detection system, which collects various behavior videos of a driver to judge by adopting a neural network method. The method adopts a neural network to distinguish from a deep learning method adopted by the invention, and the judgment result of the dangerous behavior of the method is obtained by calculation.
2) A method and a device (CN 106941602B) for recognizing the driver's behavior of a locomotive are disclosed, which predefine multiple types of behaviors of the driver and adopt a deep learning algorithm to recognize several types of daily operations of the driver. The invention adopts a Mask R-CNN deep learning method to detect the body posture of a driver and judge the danger through calculation.
3) A monitoring video-based early dangerous behavior detection method (CN 111368743A) for a deep water area of a natatorium discloses a monitoring video-based early dangerous behavior detection method for the deep water area of the natatorium, which detects that the head of a swimmer is in a swimming state or an upright state by collecting a swimming video. The invention adopts a Mask R-CNN method to detect the upper half of the forklift driver and calculates and judges the dangerous state.
Disclosure of Invention
In order to solve the technical problems, the invention aims to provide a dangerous behavior detection method and system for forklift driver practical operation examination and coaching.
The purpose of the invention is realized by the following technical scheme:
the dangerous behavior detection method for forklift driver practical operation examination and coaching comprises the following steps:
a, mounting an industrial camera at the front end of a forklift, adjusting the angle and the focal length of the camera until a waist safety belt and an upper half part of a forklift driver can be clearly shot, and recording the adjusted angle of the industrial camera;
b, capturing the position image according to the position of the forklift driver, acquiring the position pixel coordinates of the forklift driver by using a Mask R-CNN target recognition and image segmentation frame, and calculating the position angle of the upper half of the driver;
c, obtaining a proportion of a pixel interval to an actual interval according to an imaging structure of the camera system, and calculating a dangerous behavior judgment standard;
and D, shooting and monitoring the posture of a forklift driver by an industrial camera, detecting whether a safety belt is fastened, identifying the pixel coordinates of the posture of the driver by using Mask R-CNN, calculating a safety score, and judging whether the behavior is dangerous.
The dangerous behavior detection method and system for forklift driver practical operation examination and coaching comprise the following steps: the system comprises an industrial camera, an industrial camera clamp, an upper computer and a loudspeaker; the above-mentioned
The industrial camera is used for capturing a driver posture position image and uploading the image to the upper computer;
the industrial camera clamp is used for mounting an industrial camera and adjusting the angle to fix the camera;
the upper computer is used for identifying the captured driver posture position image and calculating and judging whether the driver is in dangerous behavior;
and the loudspeaker is used for playing reminding audio after dangerous behaviors of the driver are detected.
One or more embodiments of the present invention may have the following advantages over the prior art:
by adopting the industrial camera, the upper computer and the loudspeaker, the posture image of the forklift driver can be captured in real time, the pixel coordinate information of the position where the driver is located can be identified and calculated through the upper computer, the behavior of the driver can be detected and judged, and the driver can be broadcasted through the loudspeaker. The method has high automation degree, high speed and high alignment precision, can be applied to detecting and reminding dangerous behaviors of a driver of the forklift in the actual operation examination and coaching process, and has practical significance and popularization value.
The intelligent identification and detection of dangerous behaviors of a driver in the actual operation examination and coaching process of the forklift driver are realized by using a Mask R-CNN target identification and image segmentation method.
Drawings
FIG. 1 is a flow chart of a dangerous behavior detection method for forklift driver performance assessment and coaching;
FIG. 2 is a block diagram of a dangerous behavior detection method for forklift driver performance assessment and coaching.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings.
As shown in FIG. 1, the method for detecting dangerous behaviors of a forklift driver practice test and a coach comprises the following steps:
step 10, mounting an industrial camera at the front end of a forklift, adjusting the angle and the focal length of the camera until a waist safety belt and an upper half part of a forklift driver can be shot clearly, and recording the adjusted angle of the industrial camera;
step 20, capturing the position image according to the position of the forklift driver, acquiring the pixel coordinates of the position of the forklift driver by using a Mask R-CNN target recognition and image segmentation frame, and calculating the position angle of the upper half of the driver;
step 30, obtaining the proportion of the pixel spacing to the actual spacing according to the imaging structure of the camera system, and calculating a dangerous behavior judgment standard;
and step 40, shooting and monitoring the posture of a forklift driver by an industrial camera, detecting whether a safety belt is fastened, identifying the pixel coordinates of the posture of the driver by using Mask R-CNN, calculating a safety score, and judging whether the behavior is dangerous.
Referring to fig. 2, the angle between the axial direction of the lens of the adjusted industrial camera determined in the step 10 and the gravity direction is α.
As shown in fig. 2, the method for calculating the pose angle of the upper body of the driver in step 20 includes:
let N0For driver seat belt body boundary pixel count (x)0,y0,z0) Reference coordinates are referenced for the seat belt location. Using the body boundary of the driver safety belt as a reference coordinate set, obtaining { (x)0i,y0i,z0i)|i＝1,2,……,N0}，And calculating the safety belt position reference coordinates:
let N1For driver seat belt body boundary pixel count (x)1,y1,z1) Reference coordinates are referenced for the seat belt location. By recognizing the shoulder posture of the driver and storing the pixel coordinate set, the { (x) can be obtained1i,y1i,z1i)|i＝1,2,……,NlAnd calculate shoulder reference coordinates: ,
and calculating an angle beta between the position and the vertical upward direction of the driver according to the safety belt position reference coordinate and the shoulder reference coordinate:
referring to fig. 2, the step 30 determines that the safety standard of the dangerous behavior is:
according to the system structure and the actual distance deltas corresponding to one pixel pitch on the camera imaging surfacel、ΔssThe risk factor is xi, so the judgment coefficient A is calculated as follows:
wherein A is1、A2、A3There are the following constraints:
A1+A2+A3＝1
referring to fig. 2, in step 40, it is determined whether the behavior is dangerous as follows:
and recognizing the posture behavior of the driver, directly judging that the driver is in the dangerous behavior if the driver recognizes that the safety belt is fastened, and continuously detecting and judging if the safety belt is fastened.
Let St1、St2、St3Scoring the driver attitude behavior to identify the driver's initial attitude and storing the set of pixel coordinates where it is located, to obtain { (x)0j,y0j,z0j)|j＝1,2,……,M0The Mask R-CNN identifies the pixel coordinate of the driver posture, and then { (x) can be obtained1j,y1j,z1j)|j＝1,2,……,M0And calculating a driver attitude behavior risk score:
according to the driver score condition, whether the driver is in a dangerous state can be judged:
if the detection judgment result indicates that the driver is in dangerous behavior, the loudspeaker broadcasts danger and please pay attention to safe audio.
Although the embodiments of the present invention have been described above, the above descriptions are only for the convenience of understanding the present invention, and are not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (6)
1. The dangerous behavior detection method for forklift driver practical operation examination and coaching is characterized by comprising the following steps of:
a, mounting an industrial camera at the front end of a forklift, adjusting the angle and the focal length of the camera until a waist safety belt and an upper half part of a forklift driver can be clearly shot, and recording the adjusted angle of the industrial camera;
b, capturing a driver pose position image according to the pose of the forklift driver, acquiring the pixel coordinates of the pose of the forklift driver by using a Mask R-CNN target recognition and image segmentation frame, and calculating the pose angle of the upper half of the driver;
c, obtaining a proportion of a pixel interval to an actual interval according to an imaging structure of the camera system, and calculating a dangerous behavior judgment standard;
and D, shooting and monitoring the posture of a forklift driver by an industrial camera, detecting whether a safety belt is fastened, identifying the pixel coordinates of the posture of the driver by using Mask R-CNN, calculating a safety score, and judging whether the behavior is dangerous.
2. The method as claimed in claim 1, wherein the adjusted angle of the industrial camera in step a is an angle between the lens axis direction and the gravity direction, and the angle is α.
3. The method for detecting dangerous behaviors oriented to practical operator assessment and coaching of drivers of forklifts as claimed in claim 1, wherein the method for calculating the pose angle of the upper body of the driver in the step B comprises:
let the number of pixels on the body boundary of the driver safety belt be N0The reference coordinate of the safety belt part is (x)0,y0,z0) (ii) a Obtaining { (x) by taking the body boundary of the safety belt of the driver as a reference coordinate set0i,y0i,z0i)|i＝1,2,……,N0And calculating the position reference coordinates of the safety belt:
let N1For driver seat belt body boundary pixel count (x)1,y1,z1) Reference coordinates are provided for the seat belt portion; to recognize the shoulder posture of the driver and store the pixel coordinate set to obtain { (x)1i,y1i,z1i)|i＝1,2,……,NlAnd calculate shoulder reference coordinates:
calculating an angle beta between the position and the posture of the driver and the vertical upward direction according to the safety belt position reference coordinate and the shoulder reference coordinate:
4. the method for detecting dangerous behaviors oriented to practical operation examination and coaching of drivers of forklifts of claim 1, wherein the safety standard for judging the dangerous behaviors in the step C is as follows:
according to the system structure and the actual distance deltas corresponding to one pixel pitch on the camera imaging surfacel、ΔssThe risk factor is xi, so the judgment coefficient A is calculated as follows:
wherein A is1、A2、A3There are the following constraints:
A1+A2+A3＝1。
5. the method for detecting dangerous behaviors oriented to driver practice assessment and coaching of forklift truck as claimed in claim 1, wherein in the step D, it is determined whether the behaviors are dangerous as follows:
recognizing the posture behavior of a driver, directly judging that the driver is in dangerous behavior if the driver recognizes that the safety belt is fastened, and continuously detecting and judging if the safety belt is fastened;
let St1、St2、St3Scoring the driver attitude behavior; to recognize the initial posture of the driver and store the pixel coordinate set to obtain { (x)0j,y0j,z0j)|j＝1,2,……,M0And (5) identifying the pixel coordinates of the driver posture by Mask R-CNN to obtain { (x)1j,y1j,z1j)|j＝1,2,……,M0And calculating a driver posture behavior risk score:
judging whether the driver is in a dangerous state or not according to the score condition of the driver:
if the detection judgment result indicates that the driver is in dangerous behavior, the loudspeaker broadcasts danger and please pay attention to safe audio.
6. The dangerous behavior detection system for forklift driver practical operation examination and coaching is characterized by comprising an industrial camera, an industrial camera clamp, an upper computer and a loudspeaker; the above-mentioned
The industrial camera is used for capturing a driver posture position image and uploading the image to the upper computer;
the industrial camera clamp is used for mounting an industrial camera and adjusting the angle to fix the camera;
the upper computer is used for identifying the captured driver posture position image and calculating and judging whether the driver is in dangerous behavior;
and the loudspeaker is used for playing reminding audio after dangerous behaviors of the driver are detected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011388071.5A CN112541413B (en) | 2020-11-30 | 2020-11-30 | Dangerous behavior detection method and system for forklift driver real operation assessment and coaching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011388071.5A CN112541413B (en) | 2020-11-30 | 2020-11-30 | Dangerous behavior detection method and system for forklift driver real operation assessment and coaching |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112541413A true CN112541413A (en) | 2021-03-23 |
| CN112541413B CN112541413B (en) | 2024-02-23 |
Family
ID=75015473
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011388071.5A Active CN112541413B (en) | 2020-11-30 | 2020-11-30 | Dangerous behavior detection method and system for forklift driver real operation assessment and coaching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112541413B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016110374A (en) * | 2014-12-05 | 2016-06-20 | 富士通テン株式会社 | Information processor, information processing method, and information processing system |
| CN105946786A (en) * | 2016-05-06 | 2016-09-21 | 厦门理工学院 | Detection method for standard wearing of safety belt, reminding system for standard wearing of safety belt and control method for reminding system |
| CN106295601A (en) * | 2016-08-18 | 2017-01-04 | 合肥工业大学 | A kind of Safe belt detection method of improvement |
| CN108182385A (en) * | 2017-12-08 | 2018-06-19 | 华南理工大学 | A kind of pilot harness for intelligent transportation system wears recognition methods |
| CN109460699A (en) * | 2018-09-03 | 2019-03-12 | 厦门瑞为信息技术有限公司 | A kind of pilot harness's wearing recognition methods based on deep learning |
| CN109858367A (en) * | 2018-12-29 | 2019-06-07 | 华中科技大学 | The vision automated detection method and system that worker passes through support unsafe acts |
| US20190359169A1 (en) * | 2018-05-22 | 2019-11-28 | Zf Friedrichshafen Ag | Interior observation for seatbelt adjustment |
| CN111645694A (en) * | 2020-04-15 | 2020-09-11 | 南京航空航天大学 | Driver driving state monitoring system and method based on attitude estimation |
-
2020
- 2020-11-30 CN CN202011388071.5A patent/CN112541413B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016110374A (en) * | 2014-12-05 | 2016-06-20 | 富士通テン株式会社 | Information processor, information processing method, and information processing system |
| CN105946786A (en) * | 2016-05-06 | 2016-09-21 | 厦门理工学院 | Detection method for standard wearing of safety belt, reminding system for standard wearing of safety belt and control method for reminding system |
| CN106295601A (en) * | 2016-08-18 | 2017-01-04 | 合肥工业大学 | A kind of Safe belt detection method of improvement |
| CN108182385A (en) * | 2017-12-08 | 2018-06-19 | 华南理工大学 | A kind of pilot harness for intelligent transportation system wears recognition methods |
| US20190359169A1 (en) * | 2018-05-22 | 2019-11-28 | Zf Friedrichshafen Ag | Interior observation for seatbelt adjustment |
| CN109460699A (en) * | 2018-09-03 | 2019-03-12 | 厦门瑞为信息技术有限公司 | A kind of pilot harness's wearing recognition methods based on deep learning |
| CN109858367A (en) * | 2018-12-29 | 2019-06-07 | 华中科技大学 | The vision automated detection method and system that worker passes through support unsafe acts |
| CN111645694A (en) * | 2020-04-15 | 2020-09-11 | 南京航空航天大学 | Driver driving state monitoring system and method based on attitude estimation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112541413B (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110220917B (en) | Crown cap surface defect online detection method based on image processing | |
| CN107577988B (en) | Method, device, storage medium and program product for realizing side vehicle positioning | |
| CN110136449A (en) | Traffic video frequency vehicle based on deep learning disobeys the method for stopping automatic identification candid photograph | |
| CN107305627A (en) | A kind of automobile video frequency monitoring method, server and system | |
| CN106290394B (en) | System and method for detecting aluminum extrusion molding defects of CPU radiator | |
| CN113822840A (en) | Vehicle bottom inspection system and method | |
| CN107480646B (en) | Binocular vision-based vehicle-mounted video abnormal motion detection method | |
| CN113763719B (en) | Unmanned aerial vehicle-based illegal emergency lane occupation detection method and system | |
| CN106531161A (en) | Image-recognition-based apparatus and method of automatically sorting and carrying articles by mobile trolley | |
| CN111325146A (en) | Truck type and axle type identification method and system | |
| CN112084900A (en) | Underground garage random parking detection method based on video analysis | |
| CN111161312A (en) | Object trajectory tracking and identifying device and system based on computer vision | |
| Chang et al. | Real-time vehicle tracking mechanism with license plate recognition from road images | |
| CN114092683A (en) | Tire deformation identification method and device based on visual feedback and depth network | |
| CN112818834A (en) | Method, device and medium for judging avoidance of emergency vehicle at intersection | |
| CN110084171B (en) | Detection device and detection method for foreign matters on top of subway train | |
| CN112541413A (en) | Dangerous behavior detection method and system for forklift driver practical operation examination and coaching | |
| CN112950922B (en) | Fixed-point returning method for sharing electric vehicle | |
| CN114170252A (en) | Truck head anti-smashing protection system and method based on machine vision | |
| CN102043941B (en) | Dynamic real-time relative relationship identification method and system | |
| CN114119741A (en) | Shore bridge control method and device based on machine vision | |
| CN112183206B (en) | Traffic participant positioning method and system based on road side monocular camera | |
| CN111179452A (en) | ETC channel-based bus fee deduction system and method | |
| CN114694125B (en) | Data processing method for track foreign matter identification | |
| CN113205687B (en) | Drunk driving vehicle track recognition system based on video monitoring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |