CN115893136A - Elevator fault early warning method and device - Google Patents
Elevator fault early warning method and device Download PDFInfo
- Publication number
- CN115893136A CN115893136A CN202111153814.5A CN202111153814A CN115893136A CN 115893136 A CN115893136 A CN 115893136A CN 202111153814 A CN202111153814 A CN 202111153814A CN 115893136 A CN115893136 A CN 115893136A
- Authority
- CN
- China
- Prior art keywords
- vector
- abnormal behavior
- elevator
- historical
- elevator door
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 239000013598 vector Substances 0.000 claims abstract description 282
- 206010000117 Abnormal behaviour Diseases 0.000 claims abstract description 207
- 230000033001 locomotion Effects 0.000 claims abstract description 80
- 238000001514 detection method Methods 0.000 claims abstract description 78
- 230000004927 fusion Effects 0.000 claims abstract description 68
- 230000008569 process Effects 0.000 claims abstract description 22
- 238000012423 maintenance Methods 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 27
- 230000006399 behavior Effects 0.000 claims description 21
- 230000011218 segmentation Effects 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 14
- 230000007257 malfunction Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000015654 memory Effects 0.000 description 26
- 238000004590 computer program Methods 0.000 description 14
- 230000009471 action Effects 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 101150070651 Vrk1 gene Proteins 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 210000001513 elbow Anatomy 0.000 description 2
- 210000004247 hand Anatomy 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 210000003127 knee Anatomy 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007637 random forest analysis Methods 0.000 description 2
- 238000004579 scanning voltage microscopy Methods 0.000 description 2
- 210000000707 wrist Anatomy 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
- Y02B50/00—Energy efficient technologies in elevators, escalators and moving walkways, e.g. energy saving or recuperation technologies
Landscapes
- Image Analysis (AREA)
- Indicating And Signalling Devices For Elevators (AREA)
Abstract
The invention provides an elevator fault early warning method and device, and belongs to the technical field of fault early warning. The method comprises the following steps: shooting the elevator door and the passenger by using the camera to obtain image data; detecting image data to obtain the edge of the elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by using 2N X K movement speeds; detecting image data, identifying abnormal behaviors of passengers, and determining an abnormal behavior vector A in a T time period before the current moment according to the abnormal behaviors of the passengers; fusing the abnormal behavior vector A with the speed vector V to obtain a fusion vector P; inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current time; the fault category is sent to elevator maintenance personnel. The invention can early warn the elevator fault and ensure the personal safety of passengers.
Description
Technical Field
The invention relates to the technical field of fault early warning, in particular to an elevator fault early warning method and device.
Background
With the acceleration of the urbanization process, medium and high-rise buildings are more and more, so that the elevator is widely used for facilitating the travel of people, the operation safety of the elevator is increasingly important, and the safe and reliable operation of the elevator is directly related to the personal safety and the life safety of passengers.
Disclosure of Invention
The invention aims to provide an elevator fault early warning method and device, which can early warn elevator faults and ensure the personal safety of passengers.
In order to solve the above technical problem, embodiments of the present invention provide the following technical solutions:
in one aspect, an elevator fault early warning method is provided, which includes:
shooting the elevator door and passengers in the elevator by using a camera to obtain image data;
detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by utilizing 2N X K movement speeds, wherein K and N are positive integers;
detecting the image data, identifying abnormal behaviors of the rider, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the rider;
fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment;
and sending the fault category to an elevator maintenance personnel.
In an optional embodiment of the present invention, the step of detecting the image data to obtain the edge of the elevator door includes:
acquiring the depth characteristics of the image data to obtain a segmentation area of the elevator door area on the image;
and extracting the edges of the connected domains of the divided regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises coordinates of a plurality of points on the edges of the elevator door in the image.
In an alternative embodiment of the invention, determining the speed of movement of the elevator door comprises:
determining a reference point on the edge of the elevator door, and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and determining the movement speed of the elevator door according to the movement speed of the reference point.
In an alternative embodiment of the present invention, the step of detecting the image data and identifying the abnormal behavior of the occupant comprises:
detecting the image data and identifying the joint points of the passengers;
and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
In an alternative embodiment of the present invention, the step of determining an abnormal behavior vector a during a period T2 before the current time based on the abnormal behavior of the occupant includes:
establishing an abnormal behavior initial vector with the length of M, wherein M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents an abnormal behavior;
counting the number of each abnormal behavior of the passengers in a T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
In an optional embodiment of the present invention, the step of fusing the abnormal behavior vector a and the velocity vector V to obtain a fusion vector P includes any one of the following steps:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and after the dimension of the velocity vector V is reduced, the velocity vector V is spliced with the abnormal behavior vector A to obtain a fusion vector P.
In an optional embodiment of the present invention, before the step of inputting the fusion vector P into a pre-trained fault detection model, the method further includes a step of obtaining the fault detection model by training, and the step of obtaining the fault detection model by training includes:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of a historical time period according to the edge, and forming a historical speed vector V1 by using 2N x K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fused vector P1;
and obtaining the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
The embodiment of the invention also provides an elevator fault early warning device, which comprises:
the shooting module is used for shooting the elevator door and passengers in the elevator by using the camera to obtain image data;
the first detection module is used for detecting the image data to obtain the edge of the elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by using 2N x K movement speeds, wherein K and N are positive integers;
the second detection module is used for detecting the image data, identifying abnormal behaviors of passengers and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the passengers;
the fusion module is used for fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
the prediction module is used for inputting the fusion vector P into a fault detection model trained in advance and outputting the fault type within a T1 time period after the current moment;
and the fault management module is used for sending the fault types to elevator maintenance personnel.
In an optional embodiment of the invention, the first detecting module comprises:
the segmentation area acquisition unit is used for acquiring the depth characteristics of the image data to obtain the segmentation area of the elevator door area on the image;
and the divided area processing unit is used for extracting the edges of the connected areas of the divided areas to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
In an optional embodiment of the invention, the first detecting module comprises:
the first calculation unit is used for determining a reference point on the edge of the elevator door and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and the second calculating unit is used for determining the movement speed of the elevator door according to the movement speed of the reference point.
In an optional embodiment of the invention, the second detecting module comprises:
the first processing unit is used for detecting the image data and identifying the joint point of the passenger;
and the second processing unit is used for inputting the coordinates of the joint points in the continuous multi-frame images into the behavior recognition model and outputting the abnormal behavior of the passenger.
In an optional embodiment of the invention, the second detecting module comprises:
the device comprises an establishing unit, a judging unit and a judging unit, wherein the establishing unit is used for establishing an abnormal behavior initial vector with the length of M, M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents one abnormal behavior;
and the updating unit is used for counting the number of each abnormal behavior of the passengers in the T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
In an optional embodiment of the invention, the fusion module is specifically configured to perform any one of:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
after the dimension of the abnormal behavior vector A is reduced, the abnormal behavior vector A is spliced with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
In an optional embodiment of the present invention, the system further includes a training module, configured to train to obtain the fault detection model, where the training module is specifically configured to:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fused vector P1;
and obtaining the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
The embodiment of the invention also provides elevator fault early warning equipment, which comprises:
a processor; and
a memory having computer program instructions stored therein,
wherein the computer program instructions, when executed by the processor, cause the processor to perform the steps in the elevator fault pre-warning method as described above.
The embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the processor executes the steps in the elevator fault early warning method as described above.
The embodiment of the invention has the following beneficial effects:
among the above-mentioned scheme, the fault detection model of having trained in advance utilizes the fault detection model of having trained in advance to predict the fault type of elevator in a period of time in the future, sends the fault type for elevator maintainer to elevator maintainer takes relevant measure to deal with, and this embodiment can carry out the early warning to elevator trouble, with the personal safety who guarantees the person of taking person.
Drawings
Fig. 1 is a flow chart of an elevator fault early warning method according to an embodiment of the invention;
FIG. 2 is a system block diagram of an application of an embodiment of the present invention;
FIGS. 3 and 4 are schematic diagrams of data transfer according to embodiments of the present invention;
FIG. 5 is a schematic flow chart of training a fault detection model according to an embodiment of the present invention;
fig. 6 is a block diagram of the structure of an elevator fault early warning device according to an embodiment of the present invention;
fig. 7 is a block diagram showing the construction of an elevator malfunction alerting device according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages to be solved by the embodiments of the present invention clearer, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.
The embodiment of the invention provides an elevator fault early warning method and device, which can be used for early warning elevator faults and ensuring the personal safety of passengers.
Example one
An embodiment of the present invention provides an elevator fault early warning method, as shown in fig. 1, the embodiment includes:
step 101: shooting the elevator door and passengers in the elevator by using a camera to obtain image data;
fig. 2 shows a system block diagram applied in the embodiment, and the technical solution of the embodiment can perform failure early warning on a plurality of elevators. A camera is arranged in the car of each elevator, the camera can shoot the elevator door and passengers in the elevator to obtain video images, namely image data, and the acquired image data is transmitted to the data management module; in addition, each elevator is also internally provided with an elevator central control, and the elevator central control can collect the fault information of the elevator and transmit the fault information of the elevator to the data management module.
The camera can collect black and white pictures and also can collect color pictures.
Step 102: detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by utilizing 2N X K movement speeds, wherein K and N are positive integers;
in particular, elevator doors may be detected using, but not limited to, a deep learning algorithm. For example, the elevator door is detected by using MaskRCNN, image data acquired by a camera is input, the depth characteristics of the image data are acquired, the segmentation mark of the elevator door region on the image can be obtained, and the segmentation region is determined according to the segmentation mark; and extracting the edges of the connected regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises coordinates of a plurality of points on the edge of the elevator door in the image, for example, a coordinate system is established on a plane where the image is located, and by taking one point as a coordinate origin, coordinates (x 1, y 1), (x 2, y 2), (x 3, y 3), …, (xk, yk) of k points on the edge of the elevator door in the image can be obtained, and k is a positive integer. The edge coordinate chain table can record the coordinates of all points on the edge of the elevator door and can also record the coordinates of part of the points.
The general elevator doors are two, if the x-axis direction of the coordinate system is the horizontal direction, the elevator door can be distinguished to be a left elevator door or a right elevator door according to the abscissa of the point in the edge coordinate linked list. For example, the points on the edge of the elevator door can be divided into two groups according to the size of the abscissa, the point with the smaller abscissa is on the edge of the left elevator door, and the point with the larger abscissa is on the edge of the right elevator door.
Determining a reference point on the edge of the elevator door, and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images; and determining the movement speed of the elevator door according to the movement speed of the reference point.
For example, the extreme edge points of the left and right elevator doors can be used as reference points, that is, the points with the maximum and minimum x coordinates on the elevator edge can be used as reference points, the motion speed of the reference points is determined according to the position change condition of the reference points in the continuous multi-frame images, the motion speed Vi of the reference points is the moving pixel distance/time, for example, the moving pixel distance of the reference points in the two adjacent frame images is 5 pixels, the interval between the two adjacent frame images is 1/60 second, and the motion speed of the reference points is 300 pixels/second.
If it is not accurate enough to measure the moving speed of the elevator door only by the moving speed of the reference point, when the elevator door is fully closed, the number L of width pixels of the elevator door can be obtained, for example, on the same y coordinate, L = the maximum abscissa xmax-the minimum abscissa xmin of the edge point of the elevator door. The movement speed of the elevator door can be calculated to be Vr = Vi/L through the movement speed of L and the reference point
When the elevator door is fully opened, the position of the inner edge of the elevator door is marked as 0; when the elevator door is fully closed, the position of the inner edge of the elevator door is marked as 1. Extracting N position points in the range of [0,1], recording the movement speed of the elevator door reaching each position point, thus, 2N movement speeds can be recorded in each opening and closing of the elevator door, namely in one reciprocating process of the elevator door, forming a speed vector with the length of 2N, and the data form is as follows: vr1, vr2, …, vrN, … and Vr2N.
Tracing K gate opening and closing events forward from the current moment, and forming a speed vector V by utilizing 2N x K movement speeds:
Vr11、Vr12、…、Vr1N、…、Vr12N、Vr21、Vr22、…、Vr22N、…、VrK1、…、VrK2N。
k and N are positive integers and can be set as required.
Step 103: detecting the image data, identifying abnormal behaviors of passengers, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the passengers;
specifically, the image data may be detected, and the articulation point of the occupant may be identified; and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
The enclosing rectangular frame of the pedestrian in the image can be obtained through a pedestrian detection algorithm, and common algorithms include a Yolo series, an SSD and the like. After the enclosing rectangular frame of the pedestrian is obtained, sequential image analysis can be performed on each pedestrian to identify abnormal actions or behaviors of the pedestrian, such as: kicking the elevator door, extending the hands to block the elevator door, fighting near the elevator door, and the like. Specifically, the openpos algorithm may be used to obtain information of joint points of a human body, where the joint points of the human body include elbows, wrists, knees, and the like, and then a behavior recognition model may be used to input coordinate vectors of each joint point in consecutive multi-frame images as a time sequence, so as to obtain recognition results of actions and behaviors, and the behavior recognition model may adopt an LSTM algorithm.
Establishing an abnormal behavior initial vector with the length of M, wherein M is the number of the types of abnormal behaviors, and each element of the abnormal behavior initial vector represents one abnormal behavior; counting the number of each abnormal behavior of the passengers in a T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A. Assuming that the types of actions and behaviors are M, and within a time period T2 (for example, one day, set as needed) before the current time, a certain type of action behavior occurs once, the value of the corresponding element is increased by 1, so that an abnormal behavior vector a can be obtained: a1, A2, …, AM, the value of each element is the number of corresponding abnormal behaviors.
Step 104: fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
the abnormal behavior vector a and the velocity vector V may be fused in any one of the following ways:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
In a specific example, the fusion vector P may be: vr11, vr12, …, vr1N, …, vr12N, vr, vr22, …, vr22N, …, vrK1, …, vrK2N, A, A2, …, AM.
Step 105: inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment;
the fault detection model may use traditional machine learning methods such as SVM, decision trees, random forests, etc. Deep networks may also be used. Taking the example that the fault detection model uses a multi-layer deep network, the input of the fault detection model is a vector P, and the output is whether a fault E occurs after a period of time in the future. The value of E is 0-Q, 0 represents no fault, Q is a fault type, and the fault type comprises but is not limited to incapability of opening, incapability of closing, middle blockage, unsatisfactory opening and closing speed and the like.
And (3) using a deep learning universal training method, wherein an input vector P is input into the network, a fault type variable E is output, and performing iterative training.
Step 106: and sending the fault category to an elevator maintenance personnel.
Fig. 3 is a schematic diagram of data transmission in the present embodiment, and as shown in fig. 3, a camera captures multiple frames of images to form a video stream; the elevator door edge can be obtained by detecting the elevator door of the single-frame image, and the elevator door speed vector can be obtained by analyzing the speed of the elevator door; the pedestrian detection frame can be obtained by carrying out pedestrian detection on the single-frame image, and the action/action vector, namely the abnormal action vector, can be obtained by identifying abnormal actions and actions; and fusing the abnormal behavior vector and the speed vector, inputting the abnormal behavior vector into the fault detection model to predict the fault, and reporting the prediction result to elevator maintenance personnel.
In the embodiment, the fault detection model is trained in advance, the fault type of the elevator in a future period is predicted by utilizing the fault detection model trained in advance, and the fault type is sent to the elevator maintenance personnel, so that the elevator maintenance personnel can take relevant measures to deal with the fault type.
In this embodiment, the fault detection model needs to be trained in advance, and as shown in fig. 5, the step of obtaining the fault detection model through training includes:
historical image data is obtained for a period of time before (such as two weeks), and specifically, the elevator door can be detected by using a deep learning algorithm but not limited to the deep learning algorithm. For example, the MaskRCNN is used for detecting the elevator door, the image data acquired by the camera is input, the depth characteristics of the image data are acquired, the segmentation mark of the elevator door region on the image can be obtained, and the segmentation region is determined according to the segmentation mark; and extracting the edges of the connected regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises coordinates of a plurality of points on the edge of the elevator door in the image, for example, a coordinate system is established on a plane where the image is located, and by taking one point as a coordinate origin, coordinates (x 1, y 1), (x 2, y 2), (x 3, y 3), …, (xk, yk) of k points on the edge of the elevator door in the image can be obtained, and k is a positive integer. The edge coordinate chain table can record the coordinates of all points on the edge of the elevator door and can also record the coordinates of part of the points.
If the x-axis direction of the coordinate system is the horizontal direction, whether the elevator door is a left elevator door or a right elevator door can be distinguished according to the abscissa of a point in the edge coordinate linked list. For example, the points on the edge of the elevator door can be divided into two groups according to the size of the abscissa, the point with the smaller abscissa is on the edge of the left elevator door, and the point with the larger abscissa is on the edge of the right elevator door.
Determining a reference point on the edge of the elevator door, and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images; and determining the movement speed of the elevator door according to the movement speed of the reference point.
For example, the extreme edge points of the left and right elevator doors can be used as reference points, that is, the points with the maximum and minimum x coordinates on the elevator edge can be used as reference points, the motion speed of the reference points is determined according to the position change condition of the reference points in the continuous multi-frame images, the motion speed Vi of the reference points is the moving pixel distance/time, for example, the moving pixel distance of the reference points in the two adjacent frame images is 5 pixels, the interval between the two adjacent frame images is 1/60 second, and the motion speed of the reference points is 300 pixels/second.
If the moving speed of the elevator door is measured only by the moving speed of the reference point, it may not be accurate enough, and when the elevator door is fully closed, the number L of width pixels of the elevator door may be obtained, for example, on the same y coordinate, L = maximum abscissa xmax-minimum abscissa xmin of the edge point of the elevator door. The movement speed of the elevator door can be calculated to be Vr = Vi/L through the movement speed of L and the reference point
When the elevator door is fully opened, the position of the inner edge of the elevator door is marked as 0; when the elevator door is fully closed, the position of the inner edge of the elevator door is marked as 1. Extracting N position points in the range of [0,1], recording the movement speed of the elevator door reaching each position point, thus, 2N movement speeds can be recorded in each opening and closing of the elevator door, namely in one reciprocating process of the elevator door, forming a speed vector with the length of 2N, and the data form is as follows: vr11, vr2, …, vr1N, … and Vr12N.
Counting K gate opening and closing events in the historical time period, and forming a historical speed vector V1 by utilizing 2N x K movement speeds:
Vr111、Vr112、…、Vr11N、…、Vr112N、Vr121、Vr122、…、Vr122N、…、Vr1K1、…、Vr1K2N。
specifically, the image data may be detected, and the joint points of the occupant may be identified; and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
The enclosing rectangular frame of the pedestrian in the image can be obtained through a pedestrian detection algorithm, and common algorithms include a Yolo series, an SSD and the like. After the bounding rectangular frame of the pedestrian is obtained, sequential image analysis can be performed on each pedestrian to identify abnormal actions or behaviors of the pedestrian, such as: kicking the elevator door, extending the hands to block the elevator door, fighting near the elevator door, and the like. Specifically, the openpos algorithm may be used to obtain information of joint points of the human body, where the joint points of the human body include elbows, wrists, knees, and the like, and then a behavior recognition model may be used to input coordinate vectors of the joint points in consecutive multi-frame images as a time sequence, so as to obtain recognition results of actions and behaviors, where the behavior recognition model may use an LSTM algorithm.
Establishing an abnormal behavior initial vector with the length of M, wherein M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents an abnormal behavior; counting the number of each abnormal behavior of the passengers in the historical time period, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A. Assuming that the types of actions and behaviors are M, if a certain type of action and behavior occurs once in a historical time period, the value of the corresponding element is increased by 1, so as to obtain a historical abnormal behavior vector A1: a11, A12, …, A1M, the value of each element is the number of corresponding abnormal behaviors.
the historical abnormal behavior vector A1 and the historical speed vector V1 can be fused in any one of the following ways:
directly splicing the historical abnormal behavior vector A1 and the historical speed vector V1 to obtain a historical fusion vector P1;
reducing the dimension of a historical abnormal behavior vector A1, and then splicing the historical abnormal behavior vector A1 with a historical speed vector V1 to obtain a historical fusion vector P1;
and (4) reducing the dimension of the historical speed vector V1, and then splicing the historical speed vector V1 with the historical abnormal behavior vector A1 to obtain a historical fusion vector P1.
And 205, acquiring the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
Specifically, the vector P may be used as an input of the initial fault detection model, the historical fault category E may be used as an output of the initial fault detection model, and the initial fault detection model is subjected to iterative training, for example, the training is stopped after a preset number of iterations is reached.
Fig. 4 is a schematic diagram illustrating data transmission during training of a fault detection model in this embodiment, and as shown in fig. 4, a camera captures multiple frames of images to form a video stream; the elevator door edge can be obtained by detecting the elevator door of the single-frame image, and the elevator door speed vector can be obtained by analyzing the speed of the elevator door; the pedestrian detection frame can be obtained by carrying out pedestrian detection on the single-frame image, and the action/action vector, namely the abnormal action vector, can be obtained by identifying abnormal actions and actions; fusing the abnormal behavior vector with the speed vector to obtain a fused vector; obtaining elevator door fault records and classifying the elevator door faults; and training the fault detection initial model by using the classified fault types and the abnormal behavior vectors to obtain a fault detection model.
Example two
An embodiment of the present invention further provides an elevator fault early warning device, as shown in fig. 6, the device includes:
the shooting module 31 is used for shooting the elevator door and passengers in the elevator by using the camera to obtain image data;
the first detection module 32 is configured to detect the image data to obtain an edge of the elevator door, determine, according to the edge, motion speeds of the elevator door at 2N position points in a K round trip process before the current time, and form a speed vector V by using 2n × K motion speeds, where K and N are positive integers;
the second detection module 33 is configured to detect the image data, identify an abnormal behavior of the occupant, and determine an abnormal behavior vector a in a time period T before the current time according to the abnormal behavior of the occupant;
the fusion module 34 is configured to fuse the abnormal behavior vector a and the velocity vector V to obtain a fusion vector P;
the prediction module 35 is configured to input the fusion vector P into a fault detection model trained in advance, and output a fault type within a time period T1 after the current time;
and a fault management module 36 for sending the fault category to an elevator maintenance person.
In an alternative embodiment of the present invention, the first detecting module 32 includes:
the segmentation area acquisition unit is used for acquiring the depth characteristics of the image data to obtain the segmentation area of the elevator door area on the image;
and the divided area processing unit is used for extracting the edges of the connected areas of the divided areas to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
In an alternative embodiment of the present invention, the first detecting module 32 includes:
the first calculating unit is used for determining a reference point on the edge of the elevator door and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and the second calculating unit is used for determining the movement speed of the elevator door according to the movement speed of the reference point.
In an alternative embodiment of the present invention, the second detecting module 33 includes:
the first processing unit is used for detecting the image data and identifying the joint point of the passenger;
and the second processing unit is used for inputting the coordinates of the joint points in the continuous multi-frame images into the behavior recognition model and outputting the abnormal behavior of the passenger.
In an alternative embodiment of the present invention, the second detecting module 33 includes:
the device comprises an establishing unit, a judging unit and a judging unit, wherein the establishing unit is used for establishing an abnormal behavior initial vector with the length of M, M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents one abnormal behavior;
and the updating unit is used for counting the number of each abnormal behavior of the passengers in the T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
In an optional embodiment of the present invention, the fusion module 34 is specifically configured to perform any one of the following:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and after the dimension of the velocity vector V is reduced, the velocity vector V is spliced with the abnormal behavior vector A to obtain a fusion vector P.
In an optional embodiment of the present invention, the system further includes a training module, configured to train to obtain the fault detection model, where the training module is specifically configured to:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fused vector P1;
and obtaining the historical fault types of the elevator in the historical time period, and training the fault detection initial model by using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.
EXAMPLE III
An embodiment of the present invention further provides an elevator fault early warning device 50, as shown in fig. 7, including:
a processor 52; and
a memory 54, in which memory 54 computer program instructions are stored,
wherein the computer program instructions, when executed by the processor, cause the processor 52 to perform the steps of:
shooting the elevator door and passengers in the elevator by using a camera to obtain image data;
detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by utilizing 2N X K movement speeds, wherein K and N are positive integers;
detecting the image data, identifying abnormal behaviors of passengers, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the passengers;
fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment;
and sending the fault category to an elevator maintenance personnel.
Further, as shown in fig. 7, the elevator malfunction early warning apparatus 50 further includes a network interface 51, an input device 53, a hard disk 55, and a display device 56.
The various interfaces and devices described above may be interconnected by a bus architecture. A bus architecture may be any architecture that may include any number of interconnected buses and bridges. Various circuits of one or more Central Processing Units (CPUs), represented in particular by processor 52, and one or more memories, represented by memory 54, are coupled together. The bus architecture may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like. It will be appreciated that a bus architecture is used to enable communications among the components. The bus architecture includes a power bus, a control bus, and a status signal bus, in addition to a data bus, all of which are well known in the art and therefore will not be described in detail herein.
The network interface 51 may be connected to a network (e.g., the internet, a local area network, etc.), and may obtain relevant data from the network and store the relevant data in the hard disk 55.
The input device 53 may receive various commands input by an operator and send the commands to the processor 52 for execution. The input device 53 may include a keyboard or a pointing device (e.g., a mouse, a trackball, a touch pad, a touch screen, or the like.
The display device 56 may display the result obtained by the processor 52 executing the instructions.
The memory 54 is used for storing programs and data necessary for operating the operating system, and data such as intermediate results in the calculation process of the processor 52.
It will be appreciated that memory 54 in embodiments of the invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), or a flash memory, among others. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. The memory 54 of the apparatus and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
In some embodiments, memory 54 stores elements, executable modules or data structures, or a subset thereof, or an expanded set thereof: an operating system 541 and application programs 542.
The operating system 541 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic services and processing hardware-based tasks. The application programs 542 include various application programs such as a Browser (Browser) and the like for implementing various application services. A program implementing the method of an embodiment of the present invention may be included in the application program 542.
When the processor 52 calls and executes the application program and data stored in the memory 54, specifically, the following steps are performed: shooting the elevator door and passengers in the elevator by using a camera to obtain image data; detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by utilizing 2N X K movement speeds, wherein K and N are positive integers; detecting the image data, identifying abnormal behaviors of the rider, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the rider; fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P; inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment; and sending the fault category to an elevator maintenance personnel.
The method disclosed by the above embodiment of the present invention can be applied to the processor 52, or implemented by the processor 52. Processor 52 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 52. The processor 52 may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, configured to implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in a memory 54, and the processor 52 reads the information in the memory 54 and performs the steps of the above method in combination with the hardware thereof.
It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.
For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.
Further, the processor 52 is specifically configured to obtain a depth feature of the image data, and obtain a segmentation area of the elevator door area on the image; and extracting the edges of the connected domains of the divided regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
Further, the processor 52 is specifically configured to determine a reference point on the edge of the elevator door, and determine a movement speed of the reference point according to the position of the reference point in the continuous multi-frame image; and determining the movement speed of the elevator door according to the movement speed of the reference point.
Further, the processor 52 is specifically configured to detect the image data, and identify a joint point of the passenger; and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
Further, the processor 52 is specifically configured to establish an abnormal behavior initial vector with a length M, where M is a number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents an abnormal behavior; counting the number of each abnormal behavior of the passengers in a T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
Further, the processor 52 is specifically configured to perform any of the following:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
Further, the processor 52 is further configured to train and obtain the fault detection model, including:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fusion vector P1;
and obtaining the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
Example four
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the processor is caused to execute the following steps:
shooting the elevator door and passengers in the elevator by using a camera to obtain image data;
detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by utilizing 2N X K movement speeds, wherein K and N are positive integers;
detecting the image data, identifying abnormal behaviors of the rider, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the rider;
fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment;
and sending the fault category to an elevator maintenance personnel.
Further, the computer program, when executed by a processor, further causes the processor to perform the steps of: acquiring the depth characteristics of the image data to obtain a segmentation area of the elevator door area on the image;
and extracting the edges of the connected domains of the divided regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
Further, the computer program, when executed by a processor, further causes the processor to perform the steps of:
determining a reference point on the edge of the elevator door, and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and determining the movement speed of the elevator door according to the movement speed of the reference point.
Further, the computer program, when executed by a processor, further causes the processor to perform the steps of:
detecting the image data and identifying the joint points of the passengers;
and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
Further, the computer program, when executed by a processor, further causes the processor to perform the steps of:
establishing an abnormal behavior initial vector with the length of M, wherein M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents an abnormal behavior;
counting the number of each abnormal behavior of the passengers in a T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
Further, the computer program, when executed by a processor, further causes the processor to perform any of the following steps:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
Further, the computer program, when executed by a processor, further causes the processor to perform the steps of:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fusion vector P1;
and obtaining the historical fault types of the elevator in the historical time period, and training the fault detection initial model by using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.
The foregoing is a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should be construed as the protection scope of the present invention.
Claims (14)
1. An elevator fault early warning method is characterized by comprising the following steps:
shooting the elevator door and passengers in the elevator by using a camera to obtain image data;
detecting the image data to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip process before the current moment according to the edge, and forming a speed vector V by using 2N x K movement speeds, wherein K and N are positive integers;
detecting the image data, identifying abnormal behaviors of the rider, and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the rider;
fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
inputting the fusion vector P into a fault detection model trained in advance, and outputting the fault type within a T1 time period after the current moment;
and sending the fault category to an elevator maintenance personnel.
2. The elevator fault early warning method according to claim 1, wherein the step of detecting the image data to obtain the edge of the elevator door comprises:
acquiring the depth characteristics of the image data to obtain a segmentation area of the elevator door area on the image;
and extracting the edges of the connected domains of the divided regions to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
3. The elevator malfunction early warning method according to claim 2, wherein determining the moving speed of the elevator door includes:
determining a reference point on the edge of the elevator door, and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and determining the movement speed of the elevator door according to the movement speed of the reference point.
4. The elevator malfunction early warning method according to claim 1, wherein the image data is detected, and the step of identifying the abnormal behavior of the occupant comprises:
detecting the image data and identifying the joint points of the passengers;
and inputting the coordinates of the joint points in the continuous multi-frame images into a behavior recognition model, and outputting the abnormal behavior of the rider.
5. The elevator malfunction alerting method of claim 1, wherein the step of determining the abnormal behavior vector a within a time period T2 before the current time according to the abnormal behavior of the occupant comprises:
establishing an abnormal behavior initial vector with the length of M, wherein M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents an abnormal behavior;
counting the number of each abnormal behavior of the passengers in a T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
6. The elevator fault early warning method according to claim 1, wherein the step of fusing the abnormal behavior vector a with the speed vector V to obtain a fusion vector P comprises any one of the following steps:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
after the dimension of the abnormal behavior vector A is reduced, the abnormal behavior vector A is spliced with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
7. The elevator fault pre-warning method according to any one of claims 1-6, wherein the step of inputting the fusion vector P into a pre-trained fault detection model is preceded by the step of training the fault detection model, and the step of training the fault detection model comprises:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fused vector P1;
and obtaining the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
8. An elevator trouble early warning device, characterized in that includes:
the shooting module is used for shooting the elevator door and passengers in the elevator by using the camera to obtain image data;
the first detection module is used for detecting the image data to obtain the edge of the elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes before the current moment according to the edge, and forming a speed vector V by using 2N x K movement speeds, wherein K and N are positive integers;
the second detection module is used for detecting the image data, identifying abnormal behaviors of the passengers and determining an abnormal behavior vector A in a time period T before the current moment according to the abnormal behaviors of the passengers;
the fusion module is used for fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
the prediction module is used for inputting the fusion vector P into a fault detection model trained in advance and outputting the fault type within a T1 time period after the current moment;
and the fault management module is used for sending the fault types to elevator maintenance personnel.
9. The elevator malfunction early warning apparatus according to claim 8, wherein the first detection module includes:
the segmentation area acquisition unit is used for acquiring the depth characteristics of the image data to obtain a segmentation area of the elevator door area on the image;
and the divided area processing unit is used for extracting the edges of the connected areas of the divided areas to obtain an edge coordinate linked list of the elevator door in the image, wherein the edge coordinate linked list comprises the coordinates of a plurality of points on the edges of the elevator door in the image.
10. The elevator malfunction early warning apparatus according to claim 9, wherein the first detection module includes:
the first calculation unit is used for determining a reference point on the edge of the elevator door and determining the movement speed of the reference point according to the position of the reference point in continuous multi-frame images;
and the second calculating unit is used for determining the movement speed of the elevator door according to the movement speed of the reference point.
11. The elevator malfunction early warning apparatus according to claim 8, wherein the second detection module includes:
the first processing unit is used for detecting the image data and identifying the joint points of the passengers;
and the second processing unit is used for inputting the coordinates of the joint points in the continuous multi-frame images into the behavior recognition model and outputting the abnormal behavior of the passenger.
12. The elevator malfunction early warning apparatus according to claim 8, wherein the second detection module includes:
the device comprises an establishing unit, a judging unit and a judging unit, wherein the establishing unit is used for establishing an abnormal behavior initial vector with the length of M, M is the number of types of abnormal behaviors, and each element of the abnormal behavior initial vector represents one abnormal behavior;
and the updating unit is used for counting the number of each abnormal behavior of the passengers in the T2 time period before the current moment, and updating the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passengers to obtain an abnormal behavior vector A.
13. The elevator fault early warning device according to claim 8, wherein the fusion module is specifically configured to perform any one of:
directly splicing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P;
reducing the dimension of the abnormal behavior vector A, and then splicing the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;
and (4) reducing the dimension of the velocity vector V and then splicing the velocity vector V with the abnormal behavior vector A to obtain a fusion vector P.
14. The elevator fault early warning device according to any one of claims 8 to 13, further comprising a training module for training to obtain the fault detection model, wherein the training module is specifically configured to:
establishing a fault detection initial model;
detecting historical image data shot by a camera to obtain the edge of an elevator door, determining the movement speed of the elevator door on 2N position points in the K round-trip processes of the historical time period according to the edge, and forming a historical speed vector V1 by utilizing 2N X K movement speeds;
detecting historical image data shot by a camera, identifying abnormal behaviors of passengers, and determining a historical abnormal behavior vector A1 in the historical time period according to the abnormal behaviors of the passengers;
fusing the historical abnormal behavior vector A1 with the historical speed vector V1 to obtain a historical fused vector P1;
and obtaining the historical fault type of the elevator in the historical time period, and training the fault detection initial model by using the historical fault type and the historical fusion vector P1 to obtain the fault detection model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111153814.5A CN115893136A (en) | 2021-09-29 | 2021-09-29 | Elevator fault early warning method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111153814.5A CN115893136A (en) | 2021-09-29 | 2021-09-29 | Elevator fault early warning method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115893136A true CN115893136A (en) | 2023-04-04 |
Family
ID=86480255
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111153814.5A Pending CN115893136A (en) | 2021-09-29 | 2021-09-29 | Elevator fault early warning method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115893136A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117576491A (en) * | 2024-01-17 | 2024-02-20 | 浙江新再灵科技股份有限公司 | Elevator door fault detection method, elevator door fault occurrence rate prediction method and device |
-
2021
- 2021-09-29 CN CN202111153814.5A patent/CN115893136A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117576491A (en) * | 2024-01-17 | 2024-02-20 | 浙江新再灵科技股份有限公司 | Elevator door fault detection method, elevator door fault occurrence rate prediction method and device |
| CN117576491B (en) * | 2024-01-17 | 2024-04-26 | 浙江新再灵科技股份有限公司 | Elevator door fault detection method, elevator door fault occurrence rate prediction method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220327839A1 (en) | Movement state estimation device, movement state estimation method and program recording medium | |
| CN109410496B (en) | Intrusion early warning method and device and electronic equipment | |
| CN108446669B (en) | Motion recognition method, motion recognition device and storage medium | |
| CN111126153B (en) | Safety monitoring method, system, server and storage medium based on deep learning | |
| US20220004818A1 (en) | Systems and Methods for Evaluating Perception System Quality | |
| CN109360362A (en) | A kind of railway video monitoring recognition methods, system and computer-readable medium | |
| CN110589647A (en) | Method for real-time fault detection and prediction of elevator door through monitoring | |
| CN112071084A (en) | Method and system for judging illegal parking by utilizing deep learning | |
| CN110780356A (en) | Subway platform clearance foreign matter detecting system | |
| CN111914656A (en) | Personnel behavior detection method and device, electronic equipment and storage medium | |
| CN115893136A (en) | Elevator fault early warning method and device | |
| Cui et al. | Real-time detection method of driver fatigue state based on deep learning of face video | |
| Zhou et al. | Vision‐based approach for predicting the probability of vehicle–pedestrian collisions at intersections | |
| Kanu-Asiegbu et al. | Leveraging trajectory prediction for pedestrian video anomaly detection | |
| Guerrieri et al. | Real-time social distance measurement and face mask detection in public transportation systems during the COVID-19 pandemic and post-pandemic Era: Theoretical approach and case study in Italy | |
| Guerin et al. | Unifying evaluation of machine learning safety monitors | |
| Dickinson et al. | Road traffic monitoring using the TRIP II system | |
| CN116664518A (en) | Fire control access door closer abnormality detection method and system and electronic equipment | |
| CN110781847A (en) | Neural network action behavior recognition based method | |
| CN115871679A (en) | Driver fatigue detection method, driver fatigue detection device, electronic device, and medium | |
| CN114640807A (en) | Video-based object counting method and device, electronic equipment and storage medium | |
| JP2021086451A (en) | Generation device, data analysis system, generation method, and generation program | |
| CN111860100B (en) | Pedestrian number determining method and device, electronic equipment and readable storage medium | |
| CN117058627B (en) | Public place crowd safety distance monitoring method, medium and system | |
| CN113849663B (en) | Fault monitoring method, system and computer medium for camera collaborative knowledge map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |