CN115019257A - Forklift recognition processing method based on vision and image processing - Google Patents
Forklift recognition processing method based on vision and image processing Download PDFInfo
- Publication number
- CN115019257A CN115019257A CN202210736341.XA CN202210736341A CN115019257A CN 115019257 A CN115019257 A CN 115019257A CN 202210736341 A CN202210736341 A CN 202210736341A CN 115019257 A CN115019257 A CN 115019257A
- Authority
- CN
- China
- Prior art keywords
- forklift
- thermal imaging
- temperature
- picture
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 15
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000001931 thermography Methods 0.000 claims abstract description 34
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 239000011159 matrix material Substances 0.000 claims description 33
- 238000012549 training Methods 0.000 claims description 15
- 238000012216 screening Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 6
- 238000004519 manufacturing process Methods 0.000 abstract description 4
- 230000001769 paralizing effect Effects 0.000 abstract description 2
- 230000006399 behavior Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 206010033799 Paralysis Diseases 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000036626 alertness Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
- G06Q10/063114—Status monitoring or status determination for a person or group
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0635—Risk analysis of enterprise or organisation activities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
- G06Q50/265—Personal security, identity or safety
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Entrepreneurship & Innovation (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Mathematical Physics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Data Mining & Analysis (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Pure & Applied Mathematics (AREA)
- Computational Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Multimedia (AREA)
- Algebra (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Image Processing (AREA)
- Forklifts And Lifting Vehicles (AREA)
Abstract
The invention discloses a forklift identification processing method based on vision and image processing, which relates to the technical field of production environment monitoring and comprises the following steps: step 1, acquiring a picture shot by a thermal imaging camera in real time, and identifying whether a forklift exists in the picture; step 2, after the position of the forklift is identified, the position coordinate of the forklift is recorded, and the visible light picture image is converted into a two-dimensional coordinate V of whether the pixel contains the forklift or not according to pixel points y×z . The forklift identification processing method provided by the invention can improve the thermal imaging early warning quality in the warehouse, avoid meaningless early warning behavior caused by daily forklift operation, reduce the workload for safety management personnel and avoid the paralytic ideological production of early warningThe alarm is realized if necessary; in addition, customized safety management measures can be made for the forklift according to the method, the traveling route of the forklift can be monitored and controlled in cooperation with temperature area division, and safety operation is standardized.
Description
Technical Field
The invention relates to the technical field of production environment monitoring, in particular to a forklift identification processing method based on vision and image processing.
Background
The thermal imaging technology is a core technology for monitoring the temperature of environments, equipment and warehouses in the production environment of the existing factory, and a large number of factories adopt binocular thermal imaging cameras to detect the temperature in the environments and simultaneously achieve the effect of real-time monitoring.
Existing thermal imaging techniques lack the generality of the need for specific object (e.g., forklift) identification within the industry in specific scenarios. The forklift is a main vehicle used for warehousing and ex-warehouse of goods in a factory. The temperature of the forklift during operation can reach more than 80 ℃. The temperature monitoring and early warning temperature in most storage warehouse environments is below 80 ℃, and frequent forklift operation can cause multiple false alarms of an early warning system.
According to the representation of relevant safety personnel on site, more than 95% of temperature alarms in the storage warehouse every day are meaningless alarms caused by forklift operation, the original alarm mode reduces the accuracy of the alarms, and the high attention and the high alertness of the safety personnel on site to alarm behaviors are also unconsciously reduced, so that the safety problem is caused by the paralysis of thought. Meanwhile, the forklift itself is used as an object, monitoring and management are also needed, and safety accidents caused by the fact that the forklift fires and the like in the storage warehouse are avoided.
Disclosure of Invention
The invention mainly aims to provide a forklift identification processing method based on vision and image processing, and a method for establishing an independent temperature threshold value for special identification of a forklift to realize early warning by combining a thermal imaging technology.
The purpose of the invention can be achieved by adopting the following technical scheme:
a forklift identification processing method based on vision and image processing comprises the following steps
Step 1, acquiring a picture shot by a thermal imaging camera in real time, and identifying whether a forklift exists in the picture;
step 2, after the position of the forklift is identified, the position coordinate of the forklift is recorded, and the visible light picture image is converted into a two-dimensional coordinate V of whether the pixel contains the forklift or not according to pixel points y×z ；
Step 3, performing image superposition on the thermal imaging picture temperature matrix H corresponding to the visible light picture image and the visible light picture image matrix V, and screening out an area matrix F containing the forklift in the superposed image;
and 4, respectively screening the maximum temperature and the average temperature in the matrix F and the residual matrix H-F without the forklift, recording alarm values as the temperature of the forklift and the ambient temperature, and reporting the alarm values to an early warning alarm system.
Preferably, in the step 3, the two-dimensional coordinates H of the thermal imaging frame temperature matrix H m×n Where m and n represent the position of the pixel coordinate axis of the thermal imaging frame, t represents the temperature of the pixel position,
a visible light frame image matrix V, wherein x and y represent the position of the pixel coordinate axis of the visible light frame, 0 represents that no forklift is recognized for the pixel point, 1 represents that a forklift recognition pixel point exists,
preferably, when H and V are superimposed in step 3, the initial position of H in V is adjusted according to a lens shift parameter set in the thermal imaging camera, and then the H and V are superimposed;
recording t of pixel coordinates corresponding to 1 value in V in H in F and recording residual matrix in H as H-F;
MAX (F), AVG (F), MAX (H-F) and AVG (H-F) are respectively calculated and provided to a superior early warning system as temperature real-time data values.
Preferably, the step of increasing the recognition rate of the forklift is further included before the recognition of whether the forklift exists in the screen in the step 1, and the specific steps are as follows
Step A, acquiring a forklift data set, and collecting real-time monitoring visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data in a plurality of libraries from the sites of a plurality of factories;
and step B, extracting a plurality of visible light pictures containing forklift pictures in a manual screening and labeling mode, and according to the following steps of 8: 2, splitting the ratio into a training set and a test set, and training the model by using training set data;
step C, dividing the forklift in the picture into four characteristic areas, namely a bucket, a main body, wheels and a tail, and recording characteristic value information through binarization;
d, generating a forklift characteristic model after characteristic training, and performing test set testing, model adjustment and training data iteration to meet the requirement of identification precision;
and D, carrying the identification forklift characteristic model meeting the identification precision requirement in the step D into the thermal imaging camera in the step 1 so as to achieve the purpose of forklift image identification.
Preferably, the step C further includes performing special model processing on special scenes, such as occlusion, a small far scene, a small near part display scene, and a corner scene.
The invention has the beneficial technical effects that: the forklift identification processing method provided by the invention can improve the thermal imaging early warning quality in the warehouse, avoid meaningless early warning behavior caused by daily forklift operation, reduce the workload for safety management personnel, avoid the paralytic thought of early warning, and realize that the warning is necessary and real; in addition, customized safety management measures can be made for the forklift according to the method, the traveling route of the forklift can be monitored and controlled in cooperation with temperature area division, and safety operation is standardized.
Drawings
FIG. 1 is a schematic flow diagram of a method according to an embodiment of the invention.
Detailed Description
In order to make the technical solutions of the present invention more clear and definite for those skilled in the art, the present invention is further described in detail below with reference to the examples and the accompanying drawings, but the embodiments of the present invention are not limited thereto.
As shown in fig. 1, the forklift identification processing method based on vision and image processing provided by this embodiment includes the following steps
Step 1, acquiring pictures shot by a binocular thermal imaging camera in real time, namely acquiring images of a storage library in real time, and identifying whether a forklift exists in the pictures;
step 2, after the position of the forklift is identified, the position coordinates of the forklift are recorded, and the visible light picture image is converted into a two-dimensional coordinate V of whether the pixel contains the forklift according to pixel points y×z As shown in formula 2;
step 3, superposing the thermal imaging picture temperature matrix H corresponding to the visible light picture image with the visible light picture image matrix V, screening out an area matrix F containing the forklift in the superposed image, effectively distinguishing the temperature of the forklift and the temperature of the surrounding environment, and respectively recording the temperature as F if a plurality of forklifts exist 1 、F 2 ...；
And 4, respectively screening the highest temperature and the average temperature in the matrix F and the residual matrix H-F without the forklift as the recorded alarm values of the temperature and the environment temperature of the forklift and reporting the recorded alarm values to an early warning alarm system, wherein the system is internally provided with an environment temperature threshold and an early warning temperature threshold of the forklift respectively, and when the highest temperature of the forklift (the matrix F) is higher than the early warning temperature threshold of the forklift or the highest temperature in the matrix H-F is higher than the environment temperature threshold, the temperature is considered to be too high, and external alarm is triggered.
In the present embodiment, two-dimensional coordinates H of a thermal imaging frame temperature matrix H m×n Where m and n represent the position of the pixel coordinate axis of the thermal imaging frame, t represents the temperature of the pixel position,
a visible light frame image matrix V, wherein x and y represent the position of the pixel coordinate axis of the visible light frame, 0 represents that no forklift is recognized for the pixel point, 1 represents that a forklift recognition pixel point exists,
h and V need to be overlapped according to lens offset parameters set in the binocular thermal imaging camera, the resolution of the binocular lens is not good, data superposition calculation is carried out after the binocular lens is offset, the initial position of H in V is adjusted, the initial positions of H and V after adjustment are ensured to be the same coordinate of an actual picture, and then overlapping is carried out;
recording a t value of a pixel coordinate corresponding to the 1 value in V contained in H in F, and recording a residual matrix in H as H-F;
MAX (F), AVG (F), MAX (H-F) and AVG (H-F) are respectively calculated and provided to a superior early warning system as temperature real-time data values.
In the present embodiment, the two-dimensional coordinate values of F and H are as follows:
setting F to correspond to H at the (3, 4) position, H-F is:
then, through observation and mean calculation, the following results are obtained:
max (F) 88, i.e. matrix F max temperature 88 ℃;
avg (F) 67, i.e. matrix F average temperature 67 ℃;
MAX (H-F) is 86, namely the highest temperature of the residual matrix H-F without the forklift is 86 ℃;
AVG (H-F) ═ 47, i.e., the average temperature of the remaining matrix without forklift H-F was 47 ℃.
In this embodiment, as shown in fig. 1, before identifying whether there is a forklift in the frame in step 1, a step of improving accuracy of identifying the forklift is further included, specifically as follows
Step A, a forklift data set is obtained, visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data are collected from a plurality of factory sites for at least 2000 hours in a warehouse in real time, and a richer database can be established through more visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data;
and step B, extracting at least 240000 visible light pictures containing forklift pictures in a manual screening and labeling mode, and according to the following steps of 8: 2, the training set data is used for training the model, and a plurality of visible light pictures containing the forklift are divided into the training set and the testing set, so that the initial establishment of the recognition model is facilitated, and the accuracy of the model is tested;
step C, dividing the forklift in the picture into four large characteristic areas, namely a bucket, a main body, wheels and a tail, recording characteristic value information through binaryzation, and performing special model processing aiming at special scenes, such as shielding, a small far scene, a display scene of a near part and a corner scene, so that the identification accuracy of the forklift in the special scenes can be effectively improved;
d, generating a forklift characteristic model after characteristic training, and performing test set testing, model adjustment and training data iteration to meet the requirement of identification precision;
and D, carrying the identification forklift characteristic model meeting the identification precision requirement in the step D into the thermal imaging camera in the step 1 so as to achieve the purpose of effectively identifying the forklift image.
In conclusion, in the embodiment, the forklift recognition processing method provided by the embodiment can improve the thermal imaging early warning quality in the warehouse, avoid meaningless early warning behaviors caused by daily forklift operation, reduce workload for safety managers, avoid paralysis thought of early warning, and realize that warning is necessary; in addition, customized safety management measures can be made for the forklift according to the method, the traveling route of the forklift can be monitored and controlled in cooperation with temperature area division, and safety operation is standardized.
The above description is only for the purpose of illustrating the present invention and is not intended to limit the scope of the present invention, and any person skilled in the art can substitute or change the technical solution of the present invention and its conception within the scope of the present invention.
Claims (5)
1. A forklift identification processing method based on vision and image processing is characterized in that: comprises the following steps
Step 1, acquiring a picture shot by a thermal imaging camera in real time, and identifying whether a forklift exists in the picture;
step 2, after the position of the forklift is identified, the position coordinate of the forklift is recorded, and the visible light picture image is converted into a two-dimensional coordinate V of whether the pixel contains the forklift or not according to pixel points y×z ；
Step 3, performing image superposition on the thermal imaging picture temperature matrix H corresponding to the visible light picture image and the visible light picture image matrix V, and screening out an area matrix F containing the forklift in the superposed image;
and 4, respectively screening the highest temperature and the average temperature in the matrix F and the residual matrix H-F without the forklift as the temperature of the forklift and the ambient temperature, recording alarm values and reporting to an early warning alarm system.
2. The forklift identification processing method based on vision and image processing according to claim 1, characterized in that: in the step 3, the thermal imaging picture temperature matrixTwo-dimensional coordinates H of H m×n Where m and n represent the position of the pixel coordinate axis of the thermal imaging frame, t represents the temperature of the pixel position,
a visible light frame image matrix V, wherein x and y represent the position of the pixel coordinate axis of the visible light frame, 0 represents that no forklift is recognized for the pixel point, 1 represents that a forklift recognition pixel point exists,
3. the forklift identification processing method based on vision and image processing according to claim 2, characterized in that: adjusting the H to the initial position in the V for superposition according to a lens offset parameter set in the thermal imaging camera during superposition of the H and the V in the step 3;
recording t of pixel coordinates corresponding to 1 value in V in H in F and recording residual matrix in H as H-F;
MAX (F), AVG (F), MAX (H-F) and AVG (H-F) are respectively calculated and provided to a superior early warning system as temperature real-time data values.
4. The forklift identification processing method based on vision and image processing according to claim 1, characterized in that: the step of increasing the recognition of the forklift is further included before the recognition of whether the forklift exists in the picture in the step 1, which is specifically as follows
Step A, acquiring a forklift data set, and collecting real-time monitoring visible light pictures, corresponding thermal imaging pictures and thermal imaging temperature dot matrix data in a plurality of libraries from the sites of a plurality of factories;
and step B, extracting a plurality of visible light pictures containing forklift pictures in a manual screening and labeling mode, and according to the following steps of 8: 2, splitting the ratio into a training set and a test set, and training the model by using training set data;
step C, dividing the forklift in the picture into four characteristic areas, namely a bucket, a main body, wheels and a tail, and recording characteristic value information through binarization;
d, generating a forklift characteristic model after characteristic training, and performing test set testing, model adjustment and training data iteration to meet the requirement of identification precision;
and D, carrying the identification forklift characteristic model meeting the identification precision requirement in the step D into the thermal imaging camera in the step 1 so as to achieve the purpose of forklift image identification.
5. The forklift identification processing method based on vision and image processing according to claim 1, characterized in that: and the step C also comprises the step of performing special model processing on special scenes such as occlusion, a small far scene, a small near part display scene and a corner scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210736341.XA CN115019257A (en) | 2022-06-27 | 2022-06-27 | Forklift recognition processing method based on vision and image processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210736341.XA CN115019257A (en) | 2022-06-27 | 2022-06-27 | Forklift recognition processing method based on vision and image processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115019257A true CN115019257A (en) | 2022-09-06 |
Family
ID=83077564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210736341.XA Pending CN115019257A (en) | 2022-06-27 | 2022-06-27 | Forklift recognition processing method based on vision and image processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115019257A (en) |
-
2022
- 2022-06-27 CN CN202210736341.XA patent/CN115019257A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111967393B (en) | Safety helmet wearing detection method based on improved YOLOv4 | |
| US7242295B1 (en) | Security data management system | |
| CN104966304B (en) | Multi-target detection tracking based on Kalman filtering and nonparametric background model | |
| US7342489B1 (en) | Surveillance system control unit | |
| CN112560816A (en) | Equipment indicator lamp identification method and system based on YOLOv4 | |
| CN110047092B (en) | multi-target real-time tracking method in complex environment | |
| CN112184773A (en) | Helmet wearing detection method and system based on deep learning | |
| JP3486229B2 (en) | Image change detection device | |
| KR20230008612A (en) | Method for warehouse storage-location monitoring, computer device, and non-volatile storage medium | |
| WO2019043406A1 (en) | Anomaly detection from video data from surveillance cameras | |
| CN110703760A (en) | Newly-increased suspicious object detection method for security inspection robot | |
| CN116229668A (en) | Fire disaster positioning system based on infrared video monitoring | |
| CN116259002A (en) | Human body dangerous behavior analysis method based on video | |
| CN113111771A (en) | Method for identifying unsafe behaviors of power plant workers | |
| CA3081967A1 (en) | Method and system for connected advanced flare analytics | |
| CN115690496A (en) | Real-time regional intrusion detection method based on YOLOv5 | |
| KR20160093253A (en) | Video based abnormal flow detection method and system | |
| CN112422909B (en) | Video behavior analysis management system based on artificial intelligence | |
| CN103020587B (en) | Based on video image analysis flame regarding figure analysis method | |
| CN116109954B (en) | House potential safety hazard identification method, device and storage medium | |
| CN113487166A (en) | Chemical fiber floating filament quality detection method and system based on convolutional neural network | |
| CN113095160A (en) | Power system personnel safety behavior identification method and system based on artificial intelligence and 5G | |
| CN111723725A (en) | Multi-dimensional analysis system based on video AI | |
| CN115019257A (en) | Forklift recognition processing method based on vision and image processing | |
| CN115719464A (en) | Water meter durability device water leakage monitoring method based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |