CN112801072A - Elevator non-flat-layer door opening fault recognition device and method based on computer vision - Google Patents
Elevator non-flat-layer door opening fault recognition device and method based on computer vision Download PDFInfo
- Publication number
- CN112801072A CN112801072A CN202110400660.9A CN202110400660A CN112801072A CN 112801072 A CN112801072 A CN 112801072A CN 202110400660 A CN202110400660 A CN 202110400660A CN 112801072 A CN112801072 A CN 112801072A
- Authority
- CN
- China
- Prior art keywords
- elevator
- acceleration
- door opening
- sill
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/141—Control of illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
Abstract
The invention relates to a computer vision-based elevator non-flat layer door opening fault recognition device and a computer vision-based elevator non-flat layer door opening fault recognition method, wherein the device comprises a video acquisition module for acquiring an elevator sill image; the acceleration monitoring module is used for monitoring the running acceleration of the elevator so as to pre-judge whether the elevator enters a deceleration and elevator stopping state; the model training module comprises a sill slot target detection model training unit and a non-flat layer door opening classification model training unit and is used for training a model; the detection and identification module comprises an image illumination self-adaptive correction unit, a sill groove detection unit, an image angle self-adaptive correction unit and a non-flat layer door opening identification unit. In the scheme, when the device finds that the elevator is ready to stop, a detection and identification model is loaded, target images of a car door and a landing sill slot where the car door is located are analyzed, the door opening state and the door opening amplitude of the elevator are detected, and the non-flat-layer door opening fault of the elevator can be detected and identified in real time, efficiently and accurately by performing two categories of flat-layer door opening and non-flat-layer door opening on the target images.
Description
Technical Field
The invention relates to the field of elevator fault recognition, in particular to an elevator non-flat layer door opening fault recognition device and method based on computer vision.
Background
In recent years, the number of elevators in China is rapidly increased, and corresponding elevator accidents are more and more, wherein the elevator door system fault is the main reason causing the elevator accidents. There are many kinds of elevator door system failures, mainly include: the non-flat door opening (the non-flat door opening means that the absolute value of the height difference of the car door sill relative to the landing door sill is larger than 10mm, and the non-flat door opening means that the car door and the landing door are opened at the non-flat position of the elevator), non-synchronous door opening, door opening and closing failure, door clamping failure prevention and the like. The non-flat-layer door opening fault is one of the most main reasons for casualties and is also one of the most difficult problems to be thoroughly solved by the modern elevator safety guarantee technology. Therefore, the method has important practical significance for detecting and identifying the non-flat layer door opening.
The traditional non-flat-layer door opening fault detection method mainly depends on manual maintenance and troubleshooting, and the mode is low in efficiency, and influences on safe operation of the elevator due to the fact that fault hidden dangers are found for too long time.
In recent years, with the development of computer technology, some researches have attempted to realize the identification of elevator opening and closing doors through computer vision technology.
Such as: the invention has the following patents: the method and the system for detecting the elevator flat layer fault (CN 201710666494) provide a method and a system for detecting the elevator flat layer fault, when an elevator car door is opened, a target image of an area where a sill in the elevator car and a landing sill of a floor where an elevator is located are obtained, a visual angle difference value between the sill in the elevator car and the landing sill in the target image is calculated, and when the visual angle difference value is larger than or smaller than a preset value, the elevator flat layer fault is judged.
The method for identifying whether the elevator is flat or not by using the image used by the patent has the following problems:
firstly, a target image is difficult to obtain, a camera is arranged at the upper part in the car, and a sill or a door leaf is easily shielded by passengers at the entrance and exit of the car;
secondly, the image detection method is difficult to realize, and a method of viewing the intersection value of the layer sill and the landing sill is adopted, so that if the landing sill is positioned at a lower position during door opening, the landing sill is shielded, and the algorithm is invalid;
thirdly, the image recognition efficiency and accuracy are low, the elevator door and the sill have various specifications, and the currently adopted image recognition technology is simple and cannot quickly and accurately recognize the target image;
fourth, the fault identification process occurs during the entire elevator operation, which results in unnecessary computational effort.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a computer vision-based elevator non-flat layer door opening fault recognition device and method, which can monitor the elevator non-flat layer door opening fault in real time on line, reduce the manual calibration workload of a non-flat layer target sample, realize the rapid training of a floor trough model and a non-flat layer model, improve the target detection accuracy of an elevator non-flat layer image and a door opening and closing image, have the image noise reduction capability in low-illumination and high-illumination environments and improve the intelligent detection and recognition capability of the whole elevator fault.
The embodiment of the invention provides a computer vision-based elevator non-flat layer door opening fault recognition device which comprises a video acquisition module, an acceleration monitoring module, a model training module and a detection recognition module, wherein the video acquisition module is used for acquiring a video signal; wherein the content of the first and second substances,
the video acquisition module comprises a camera device and is used for acquiring an elevator sill image; the video acquisition module is electrically connected with the detection identification module; the lens of the camera device is arranged at a gap between the elevator car door and the hall door and is vertical to and right against the sill groove so as to shoot the hall door sill groove and the car door sill groove simultaneously;
the acceleration monitoring module comprises an acceleration sensor and a microprocessor; the acceleration monitoring module is used for monitoring the running acceleration of the elevator so as to pre-judge whether the elevator enters a deceleration and elevator stopping state, and the acceleration monitoring module is electrically connected with the detection and identification module;
the model training module comprises a sill slot target detection model training unit and a non-flat layer door opening classification model training unit; the sill trough target detection model training unit is used for pre-training a sill trough detection model for loading and using by the detection and identification module; the non-flat-layer door opening classification model training unit is used for pre-training a non-flat-layer door opening classification model for loading and using by the detection and recognition module;
and the detection and identification module is used for carrying out self-adaptive processing on an elevator sill image and loading a model trained by the model training module for prediction and identification when the elevator enters a deceleration and elevator stopping state according to the acceleration monitoring module so as to judge whether the elevator has a non-flat-layer door opening fault.
Preferably, the microprocessor is specifically configured to perform acceleration detection on the acceleration curve after the filtering smoothing processing by using the variance, the range and the ratio of the index distance corresponding to the range to the length of the acceleration sequence to determine whether the elevator is stopped;
wherein the variance is used for measuring the fluctuation degree of the acceleration sequence, and the expression is the stability of the acceleration
Expressed as shown in equation (1):
wherein the content of the first and second substances,
indicating a sequence of accelerations
The value of each acceleration is taken as the value,
the acceleration mean value of the acceleration sequence is shown, n represents the length of the acceleration sequence, namely the acceleration sequence is a sequence consisting of n acceleration values;
the range is the maximum range for measuring the value variation, is a convenient index for measuring the value variation, is the difference between the maximum value and the minimum value, and is used for measuring the variation of the value
Expressed as shown in equation (2):
wherein the content of the first and second substances,
represents the maximum value in the sequence of accelerations,
represents the minimum value of the acceleration sequence;
the ratio of the index distance corresponding to the range to the length of the acceleration sequence is characterized by describing the degree of acceleration shock, and describing the proportion of the distance between the sequence positions of the maximum value and the minimum value to the whole running acceleration sequence, and the proportion is expressed by r, as shown in formula (3):
wherein the content of the first and second substances,
indicating the position in the sequence of the maximum acceleration in the sequence,
then the position of the minimum acceleration is shown, and the sequence position is shown as a few points; n is the length of the whole acceleration sequence;
when the variance is more than 1000, the range is more than 500 and/or the ratio of the index distance corresponding to the range to the length of the acceleration sequence is less than 10%, the elevator is considered to be about to stop; otherwise, the microprocessor judges that the acceleration is normal and the elevator is in normal operation.
Preferably, the acceleration sensor has a plurality of acquisition time intervals; the accuracy of the judgment of the microprocessor under each acquisition time interval is calculated by traversing the acquisition time interval of the acceleration sensor, the ratio of the accuracy to the information quantity acquired by the acceleration sensor in unit time is calculated, and when the ratio reaches the maximum value, the current corresponding acquisition time interval is set as the default acquisition time interval.
Preferably, the training method of the sill-trough detection model training unit in the model training module includes the following steps:
step 41: manufacturing a sill groove detection training set, and labeling a car picture containing a sill groove by using LabelImg, specifically framing out a sill groove image and labeling the type;
step 42: building a target detection model structure, using a model trained on an ILSVRC data set as an initial weight of a network backbone by adopting a transfer learning method, randomizing other parameters except the initial weight, and training on the sill-slot detection training set built in the step 41, wherein a loss function comprises a class classification loss function loss _ cls and a sill-slot boundary frame position return loss function loss _ bbox;
step 43: and setting a stopping strategy, and stopping training when the specified iteration times are reached or the prediction precision of the model in the verification set reaches a set value.
Preferably, the non-flat layer door opening image classification model training unit in the model training module is used for training a non-flat layer door opening recognition model and performing secondary classification on the sill groove images to recognize a flat layer state;
the method for training the non-flat layer door opening classification model comprises the following steps:
step 51: the method comprises the steps of manufacturing a sill groove image classification training set, wherein in a door opening state, a sill groove image with the absolute value of the height difference between a hoistway door sill groove and a car door sill groove being less than or equal to 10mm is collected and used as a flat-layer door opening data set, and the rest sill groove image is a non-flat-layer door opening data set;
step 52: downloading and loading an inclusion V3 model as a pre-training model;
step 53: executing a transfer learning strategy, removing an output full connection layer and a softmax layer, adding a new two-classification full connection layer and a softmax classification layer, adding and deleting a preset intermediate network layer as an improved depth convolution neural network based on an initiation structure; the deep convolution neural network contains an inclusion structure, and different scale information of the image is extracted through multi-convolution kernel fusion to obtain better representation, wherein the inclusion structure contains a sparse network structure with 1 × 1 convolution to reduce dimension and build less parameters; the predetermined intermediate network layer comprises an initiation block, a 1 × 1 convolution;
step 54: loading the deep convolutional neural network obtained in the step 53, and creating a training script file;
step 55: preprocessing the sample of the classification training set of the sill-trough image, inputting the trained parameters of batch _ size, epoch and iteration, wherein the batch _ size represents the number of required samples used in one parameter updating process, 1 iteration is equal to 1 time of training by using the batch _ size samples, and the epoch is the total iteration times of all the samples;
step 56: setting a stopping strategy, and stopping training when iteration reaches a specified epoch number and iteration number or when the prediction precision of the model in the verification set reaches a set value;
and 57: a checkpoint mechanism is additionally arranged, and when the optimal value is reached in the process of training each epoch model, a model file is output;
step 58: and performing data preprocessing according to the batch _ size loading sample and then executing a training process.
Preferably, training samples in the sill-trough image classification training set are obtained by performing random clipping, turning and feature transformation on original samples; the feature transformation is shown in the publications (4), (5) and (6):
wherein the content of the first and second substances,
representing a central pixel, for a given central point
The position of the circular neighborhood pixel is
, 
Is the radius of the sample, and,
is shown as
A number of sample points are sampled at the time of sampling,
is the number of samples to be taken,
if the difference value between the adjacent position pixel and the middle position central pixel is positive number or 0, returning to 1; otherwise, the value is 0, and the value is,
is a function of the rotation of the rotating body,
show that
Right circulation
A bit.
Preferably, the image illumination adaptive correction unit in the detection and identification module is specifically configured to perform illumination adaptive correction on an input image acquired by the video acquisition module, and process a sill-box image when ambient illumination is too dark and too bright, so as to improve illumination uniformity and enhance image quality, and the adaptive processing method is as shown in formulas (7) and (8):
wherein the content of the first and second substances,
is the illumination characteristic of the corrected output image,
for inputting images
Extracting illumination components, wherein m is a brightness mean value of the illumination components;
the coefficient is a self-adaptive coefficient, and the parameters k and r are variable parameters iteratively learned by a machine learning method according to the actual illumination condition of the elevator car; the sill slot detection unit in the detection identification module is used for:
loading a target detection model to detect a sill slot, and judging that a door is opened when the sill slot is detected; judging the door opening degree alpha of the elevator according to the detected length of the sill slot and the ratio of the total length of the sill slot, wherein the detected ratio of the actual length of the sill slot to the actual total length of the sill slot is equal to the corresponding pixel ratio; when alpha is 0, the elevator is not opened; when the door opening degree alpha is larger than or equal to a set threshold value, identifying and judging whether to open the door; the image angle adaptive correction unit in the detection identification module is specifically configured to perform angle transformation on the detected sill-slot image to adapt to angle distortion caused by different lenses, different types of door systems, and different types of door opening manners, as shown in the following publications (9) (10) (11) (12):
wherein
Is the coordinates of the pixel points after the transformation,
is the coordinate of the pixel point before transformation,
respectively the width and the height of the image,
is the included angle between the center of the camera lens and the horizontal central line of the sill groove,
is the included angle between the center of the camera lens and the center line of the sill groove,
and
is formed by
And
and determining vectors for measuring the rotation angle of the two-dimensional plane and the rotation degree of the three-dimensional space around the z axis, and performing self-adaptive training by taking the vectors as the hyper-parameters.
Preferably, the non-flat layer door opening recognition unit in the detection recognition module is specifically configured to load a non-flat layer door opening classification model, determine whether the hall door and the car door are non-flat layers, and give a confidence β, and determine that the hall door is open when β is greater than a set threshold.
The embodiment of the invention also provides a non-flat layer door opening fault identification method based on deep learning, which comprises the following steps:
step 101: starting a video acquisition module; the acceleration monitoring module acquires an acceleration signal of the elevator car and transmits the acceleration signal to the microprocessor; the microprocessor receives the acceleration signal, carries out filtering smoothing processing, analyzes an acceleration curve after the filtering smoothing processing is finished, and enters step 102 if the acceleration curve is judged to enter a deceleration and elevator stopping stage, and continues to carry out acceleration monitoring if the acceleration curve is judged to normally run;
step 102: the detection and identification module receives the information of the acceleration monitoring module and starts to intercept an elevator car running video according to the monitored time for preparing to stop the elevator; the detection and identification module carries out frame processing on the intercepted video, carries out illumination self-adaptive correction on each frame of image and enters step 103;
step 103: loading a acamprosate groove detection model by a detection and identification module, carrying out target detection on the image obtained in the step 102, judging that the door is opened when the acamprosate groove is detected, and entering the step 104;
step 104: the detection and identification module performs angle self-adaptive correction on the sill-slot image detected in the step 103, intelligently processes image angle distortion caused by factors such as different lenses, door systems of different models, door opening modes of different types and the like, and enters a step 105;
step 105: the detection recognition module loads a non-flat layer door opening classification model, judges whether the image processed in the step 104 is a non-flat layer door opening or not, gives a confidence coefficient beta, and judges that the non-flat layer door opening is in fault when the beta is larger than a set threshold value; otherwise, go back to step 101.
The elevator non-flat layer door opening fault recognition device and method based on computer vision provided by the invention can fully cover various types of elevator door systems, meet the requirement of rapid development of the elevator industry, realize intelligent recognition and prediction of elevator non-flat layer door opening, overcome the defects of the traditional detection mode, reduce the manual calibration workload of non-flat layer target samples, improve the target detection accuracy and recognition speed of elevator non-flat layer images and door opening and closing images, and achieve the model recognition accuracy of 95.3%. The device adopts a leading elevator fault identification technology, and realizes extremely strong compatibility.
The invention has the following beneficial effects:
1. the sensor difference of all kinds of elevators is fully considered, the detection mode of video identification is adopted, the complex problem of the original elevator sensor device is avoided being improved, and the identification device is simpler and more convenient to install.
2. The difference of the model and the specification of the door system is fully considered, a deep learning mode is utilized to train a good waistcoat model and a non-flat layer door opening model in advance, and the workload of manually calibrating non-flat layer images can be greatly reduced.
3. The distortion of the shooting angle of the lens caused by different lenses, door systems of different models, door opening modes of different types of elevators and the like is fully considered, and the image angle self-adaptive correction unit is arranged in the device and can automatically change the angle of the detected sill groove image.
4. Aiming at the influence of the complex environment illumination of the elevator on the image, the device is provided with an image illumination self-adaptive correction unit, and is particularly used for performing illumination self-adaptive correction on the input image acquired by the video acquisition module so as to improve the illumination uniformity and enhance the image quality.
5. The condition that the elevator door and the sill are easily shielded by passengers or equipment is fully considered, reasonable correspondence is achieved, the lens is arranged at a gap between the elevator car door and the hoistway door, the hoistway door sill groove and the car door sill groove can be shot simultaneously, shielding is avoided, and the opening degree of the elevator is judged according to the detected ratio of the length of the sill groove to the total length of the sill groove.
6. In order to improve the target detection accuracy and the recognition speed of the elevator non-flat-layer image and the door opening and closing image, respectively manufacturing a sill slot detection training set and a sill slot image classification training set; and training a sill groove detection model, obtaining a training non-flat layer door opening recognition model by randomly cutting, turning and carrying out characteristic transformation on an original sample, and quickly recognizing a flat layer or a non-flat layer by classifying images of the sill groove.
7. Aiming at the problem of computing power consumption of the detection identification module, the acceleration monitoring module is configured, when the elevator is found to enter a deceleration and ladder-stopping state, the detection identification module starts to work, otherwise, the detection identification module is in standby state so as to save computing power.
8. The deep learning model is adopted, the accuracy of detection and identification is guaranteed, and compared with the traditional methods of distance measurement by using a sensor and the like, the method has better robustness and applicability.
9. The method fully considers the problems in industrialization and marketization, realizes the distance measuring function by optimizing an image recognition model and an algorithm on the basis that a common monocular camera is used as an image acquisition device, has the advantages of lower cost, smaller volume, less calculation force demand, higher image processing speed and the like compared with other image distance measuring modes such as a depth camera and the like, has good industrialization, and is easier to install on site.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of an elevator non-flat layer door opening fault recognition device provided by an embodiment of the invention.
Fig. 2 is a flowchart of an elevator non-flat floor door opening fault identification method provided by an embodiment of the invention.
FIG. 3 is a diagram illustrating an example of sample feature preprocessing operation according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a working process of the detection and identification module according to the embodiment of the present invention.
Fig. 5 is a schematic view of installation positions of the video capture module and the acceleration monitoring module according to the embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of the present invention provides a computer vision-based elevator non-flat layer door opening fault recognition apparatus, which includes a video acquisition module 10, an acceleration monitoring module 20, a model training module 30, and a detection recognition module 40; wherein the content of the first and second substances,
as shown in fig. 5, in this embodiment, the video capture module 10 is disposed at the top inside the elevator car 1, the video capture module 10 may be a camera device, such as a miniature camera or a camera, and the lens 3 is disposed at the gap between the elevator car door 6 and the landing door 7, perpendicular to the sill groove, and faces the sill groove for shooting, so that the landing door sill groove 5 and the car door sill groove 4 can be shot simultaneously.
In the embodiment, the specific installation position of the lens is determined according to the type of the door of the elevator and is consistent with the position of the opened door of the elevator, for example, when the elevator is a split door, the lens is installed at the middle position of the door; when the elevator is a side door, the lens is arranged at the door opening end of the car door. In this embodiment, the camera device preferably employs an infrared wide-angle camera, which can adapt to an environment with weak light, wherein a lens of the camera is directly opposite to the ground trough.
Further, camera device can adopt light filling lamp and illuminance sensor, and when illuminance was less than the settlement limit, the light filling lamp was automatic to be opened, supplyes the illumination to the sill groove.
In this embodiment, the video capture module 10 is electrically connected to the detection identification module 40, and the video capture module 40 can transmit the video signal collected in real time to the detection identification module 40 through the connection circuit and store the video signal.
The acceleration monitoring module 20 comprises an acceleration sensor and a microprocessor, and the microprocessor can be a single chip microcomputer; the acceleration monitoring module 20 is used for monitoring the running acceleration of the elevator so as to pre-judge whether the elevator enters a deceleration state, and the acceleration monitoring module 20 is electrically connected with the detection and identification module 40.
The time interval of the acceleration sensor for acquiring the acceleration can be 60ms, 40ms and 25 ms. The microprocessor judges the elevator stopping accurately when detecting the acceleration and stores hardwareThe storage requirement and the information transmission requirement are exponentially increased; particularly, an inflection point appears at 40ms, at which the ratio of the accuracy of the judgment of the microprocessor to the information quantity acquired by the acceleration sensor in unit time reaches a maximum value, so that in the embodiment, the acceleration sensor adopts an acquisition time interval of 40ms, namely, an acquisition frequency of 25HZ, and the unit of acceleration is cm/s2。
And the microprocessor detects the acceleration of the acceleration curve after the filtering smoothing processing by using the variance, the range and the ratio of the index distance corresponding to the range to the length of the acceleration sequence so as to judge whether the elevator stops. The variance is used for measuring the fluctuation degree of the acceleration sequence, and the expression is the stability of the acceleration
Expressed as shown in equation (1):
wherein the content of the first and second substances,
indicating a sequence of accelerations
The value of each acceleration is taken as the value,
and the acceleration mean value of the acceleration sequence is shown, and n represents the length of the acceleration sequence, namely the acceleration sequence is a sequence consisting of n acceleration values.
The range is the maximum range for measuring the value variation, is a convenient index for measuring the value variation, is the difference between the maximum value and the minimum value, and is used for measuring the variation of the value
Expressed as shown in equation (2):
wherein the content of the first and second substances,
represents the maximum value in the sequence of accelerations,
representing the minimum value of the acceleration sequence.
The ratio of the index distance corresponding to the range difference to the length of the acceleration sequence is the sharp variation degree of the depicted acceleration, the ratio of the distance between the sequence positions of the maximum value and the minimum value to the whole running acceleration sequence is described, and the ratio is used
Expressed, as shown in equation (3):
wherein the content of the first and second substances,
indicating the sequence position in the sequence where the maximum acceleration is located,
the sequence position at which the minimum acceleration is located is indicated, in this example the sequence position is indicated as the first few points;
the length of the entire acceleration sequence.
In the embodiment, the variance is more than 1000, the range is more than 500, and/or the ratio of the index distance corresponding to the range to the length of the acceleration sequence is less than 10%, the elevator is considered to be about to stop; otherwise, the microprocessor judges that the acceleration is normal. When the relation among the variance, the range and the ratio of the index distance corresponding to the range to the length of the acceleration sequence is 'sum', the explanation condition is harsh, and the condition that the elevator is about to stop is judged to be more accurate, but the condition of missing inspection can occur; when the relation is "or", although the condition is relaxed and the situation that the elevator is about to stop can be comprehensively checked, the accuracy of the situation that the elevator is determined to be about to stop is lowered. The acceleration sequence length represents the length of acceleration data in a time interval from the beginning of the speed of the elevator car to the return to zero; the index distance corresponding to the range difference represents the acceleration data length between the maximum value and the minimum value of the acceleration in the acceleration sequence; the data length is the quantity of data, and the speed of elevator car is gathered by speed sensor, and speed sensor sets up in elevator car.
The model training module 30 comprises a acamprosate groove target detection model training unit and a non-flat layer door opening image classification model training unit; the method is used for pre-training the sill slot detection model and the non-flat layer door opening recognition model for loading and using of the detection module.
The detection and identification module 40 comprises an image illumination self-adaptive correction unit, a sill groove detection unit, an image angle self-adaptive correction unit and a non-flat layer door opening identification unit; the method is used for carrying out self-adaptive processing on the image and loading the trained model for prediction and identification.
As shown in fig. 2, the video capture module 10 is electrically connected to the detection and identification module 40; the model training module 30 and the detection and recognition module 40 are both disposed on the back-end server.
The training method of the sill trough detection model training unit in the model training module 30 comprises the following steps:
step 1: manufacturing a sill groove detection training set, and carrying out batch labeling on car pictures containing a sill groove by using LabelImg, specifically framing out a sill groove image and labeling the type;
step 2: and (2) building a target detection model structure, using a model trained on the ILSVRC data set as an initial weight of the network backbone by adopting a transfer learning method, randomizing other parameters except the initial weight, and training on the data set built in the step (1), wherein the loss function comprises a class classification loss function loss _ cls and a sill slot bounding box position return loss function loss _ bbox. In this embodiment, the category loss function adopts a binary cross entropy, and the border position regression loss function adopts an L2 loss function;
and step 3: and setting a stopping strategy, and stopping training when the specified iteration times are reached or the prediction precision of the model in the verification set reaches a set value.
In the present embodiment, the back bone network of the acamprosate image target detection model using YoLoV3 and YoLoV3 for feature extraction is a dark net53 network, the output layer has three full convolution networks as output layers, each output layer outputs a result on a feature map with different dimensions, and the output dimension is
Where S is the scale size of the S × S feature map, B represents the number of frames output per grid, C is the number of classes detected, and 5 is the 4-position frame coordinates output and the confidence score belonging to the object. In this embodiment, only one category of candela groove is thus 1, the input image size is 416 × 416, and ((52 × 52) + (26 × 26) + (13 × 13)) × 3=10647 bounding boxes are predicted.
In this embodiment, the data set may be generated by collecting the car picture of the elevator during the door opening period by using the camera and then labeling the car picture containing the acamprosate groove with the labeling tool label img according to the position and the type. After the data set is obtained, the data set is subjected to data cleaning and then randomly divided into a training set and a verification set according to the ratio of 8: 2. The darknet53 model trained on the ILSVRC image dataset was used as the initial weight of the network backbone, and the other parameters except the initial weight were initialized with the he normal method. The loss functions used in the training include a category loss function using binary cross entropy and a bounding box position regression loss function using an L2 loss function. Training is done on the training set and validation is done on the validation set to set the stopping strategy according to real-time accuracy.
The non-flat layer door opening image classification model training unit in the model training module 30 is used for training a non-flat layer door opening recognition model and performing two classifications on the sill groove images so as to recognize a flat layer or a non-flat layer;
the Inception structure obviously increases the number of nerve units in each step, and preprocessing polymerization is carried out on different scales, so that the manual determination of the type of a filter in a convolutional layer is replaced, and a network can learn parameters by self. Whereas inclusion V3 achieves stacking more of the inclusion modules by decomposing the convolution kernel size, changing the reduced feature map size, to increase utilization of computing resources, which amounts to 47 tiers.
The method for training the non-flat layer door opening recognition model comprises the following steps:
step 1: the method comprises the steps of manufacturing a sill groove image classification training set, wherein in a door opening state, a sill groove image with the absolute value of the height difference between a hoistway door sill groove and a car door sill groove being less than or equal to 10mm is collected and used as a flat-layer door opening data set, and the rest sill groove image is a non-flat-layer door opening data set;
step 2: downloading and loading an inclusion V3 model as a pre-training model;
and step 3: executing a transfer learning strategy, removing an output full-connection layer and a softmax layer, adding a new two-classification full-connection layer and a softmax classification layer, and optionally adding and deleting partial intermediate network layers such as partial initiation blocks and 1 multiplied by 1 convolution to serve as an improved depth convolution neural network based on an initiation structure; the network contains an inclusion structure, and different scale information of the image is extracted through multi-convolution kernel fusion to obtain better representation, and the inclusion structure contains a sparse network structure with less parameters which are constructed through 1 × 1 convolution in a dimensionality reduction mode.
And 4, step 4: loading the deep convolutional neural network obtained in the step 3, and creating a training script file;
and 5: preprocessing is carried out on samples of the classification training set of the sill-trough images, parameters of training, namely, batch _ size, epoch, iteration and the like are input, the batch _ size represents the number of required samples used in one parameter updating process, 1 iteration is equal to 1 time of training by using the batch _ size samples, and the epoch is the total iteration times of all the samples. In this embodiment, batch _ size is 64 and epoch is 100.
Step 6: and setting a stopping strategy, and stopping training when iteration reaches the specified epoch number and iteration number or the prediction accuracy of the model reaches a set value.
And 7: a checkpoint mechanism is additionally arranged, and when the optimal value is reached in each epoch deep learning process, a model file is output;
and 8: and performing data preprocessing according to the batch _ size loading sample and then executing a training process.
The Softmax function is a normalized exponential function, is a generalization of a logic function and is used for multi-classification. The Softmax function maps the outputs of a plurality of neurons into a (0,1) interval, representing the output probability, and the formula is as follows:
wherein the content of the first and second substances,
indicating the probability of inferring belonging to the second class,
the output of the previous layer, i.e. the input of softmax, is represented with dimension C.
The non-flat layer door opening image classification model training unit is characterized in that a training sample is obtained by randomly cutting, turning and carrying out characteristic transformation on an original sample; the feature transformation is shown in formulas (4), (5) and (6):
wherein the content of the first and second substances,
representing a central pixel, for a given central point
The position of the circular neighborhood pixel is
Is the radius of the sample
Is shown as
A number of sample points are sampled at the time of sampling,
is the number of samples
. If the difference value between the adjacent position pixel and the middle position pixel is positive number or 0, returning to 1; otherwise it is 0.
Is a function of the rotation of the rotating body,
show that
Right circulation
A bit. The operation process of feature transformation is shown in fig. 3.
Assume that in the present embodiment:
the total number of non-flat layer door opening samples is 800, and the total number of flat layer door opening samples is 1000;
batch _ size =64, training set: test set = 8: 2;
the confusion matrix for the test set is as follows:
y =0 represents the number of samples of the true negative case, and y =1 represents the number of samples of the true positive case; yhat =0 represents the number of samples inferred as negative, and yhat =1 represents the number of samples inferred as positive.
Wherein the total number of test sets = (800 + 1000) × 0.2=192+9+9+151= 360;
the model identification precision = (192 + 151)/360 ≈ 95.3%, and the set value requirement of the model prediction precision is met.
The image illumination self-adaptive correction unit in the detection and identification module is specifically used for performing illumination self-adaptive correction on an input image (namely a sill groove image) acquired by the video acquisition module, processing a low-illumination image of a dim elevator, or a high-illumination image of a strong lamp illumination elevator, a sightseeing elevator and the like, particularly an overexposure image and a high-reflection image caused by reflection of light of a metal sill groove, so as to improve the illumination uniformity and enhance the image quality, and the self-adaptive processing method is as shown in formulas (7) and (8):
wherein the content of the first and second substances,
is the illumination characteristic of the corrected output image,
in order to input an image, the image is,
and (5) extracting illumination components, wherein m is the brightness mean value of the illumination components. Herein, the
The coefficient is an adaptive coefficient, and the parameters k and r are variable parameters which can be iteratively learned by an available machine learning method according to the actual illumination condition of the elevator car. This example
Coefficient of available
。
A sill slot detection unit in the detection identification module 40 loads a target detection model to detect a sill slot; when the sill slot is detected, the door opening can be judged, further, the door opening degree alpha of the elevator can be judged according to the detected length of the sill slot and the ratio of the total length of the sill slot, and the ratio of the detected actual length of the sill slot and the actual total length of the sill slot is equal to the corresponding pixel ratio of the sill slot; when alpha is 0, the elevator is not opened; and when the door opening degree alpha is larger than or equal to the set threshold value, identifying the sill slot image to judge whether the door is opened in a non-flat layer. In this embodiment, the α threshold is set to 0.1.
The image angle adaptive correction unit in the detection and identification module 40 is specifically configured to perform angle transformation on the detected sill-slot image to adapt to angle distortion caused by different lenses, different door systems, different door opening manners, and the like, as shown in formulas (9) (10) (11) (12):
wherein， 
Is the coordinates of the pixel points after the transformation,
is the coordinate of the pixel point before transformation,
respectively the width and the height of the image,
is the included angle between the center of the camera lens and the horizontal central line of the sill groove,
is the included angle between the center of the camera lens and the center line of the sill groove.
And
is formed by
And
determining rotation angle of two-dimensional plane and rotation of three-dimensional space around z-axisThe vector of the degree can be used as a hyper-parameter to carry out self-adaptive training.
The non-flat layer door opening recognition unit in the detection recognition module 40 is specifically configured to load a non-flat layer door opening classification model, determine whether the hall door and the car door are non-flat layers, and give a confidence β, and determine that the hall door is open when β is greater than a set threshold. The threshold for confidence level in this embodiment is set to 90%.
During detection, the trained detection model is loaded to detect the sill slot in the image, the image is normalized and then input into the trained network to obtain output, and the output is converted to obtain the actual position boundary frame. The output conversion formula is as follows:
wherein the content of the first and second substances,
，
，
，
the frame information directly output by the network output layer,
，
， 
，
respectively the coordinates and sampling of the upper left corner of the feature map gridThe width and height of the anchor used.
，
，
，
Coordinates of the upper left corner of the frame and the width and height of the frame are calculated for the position on the actual image after conversion.
And finally, filtering the overlapped frames of the obtained frame by adopting an NMS algorithm, and reserving the final effective frame. The detected sill slot image is then used as the input of the non-flat layer door opening recognition model. The workflow of the detection recognition module is shown in fig. 4.
Referring to fig. 2, an embodiment of the present invention provides a computer vision-based elevator non-leveling fault identification method, including the following steps:
step 1: starting an image acquisition module; the acceleration monitoring module acquires an acceleration signal of the elevator car and transmits the acceleration signal to the microprocessor; the microprocessor receives the acceleration signal, carries out filtering smoothing processing, analyzes the acceleration curve after the filtering smoothing processing is finished, enters the step 2 if judging that the vehicle enters the deceleration stage and is ready to stop the vehicle, and continues to carry out acceleration acquisition if judging that the vehicle normally runs;
step 2: the detection and identification module receives acceleration information and intercepts an elevator car running video according to the time for monitoring the elevator stopping preparation according to the acceleration; performing frame processing on the intercepted video, performing illumination self-adaptive correction on each frame of image, and entering the step 3;
and step 3: loading a acamprosate groove detection model by the detection and identification module, carrying out target detection on the image obtained in the step (2), judging that the door is opened when the acamprosate groove is detected, and entering the step (4);
and 4, step 4: the detection and identification module performs angle self-adaptive correction on the sill groove image detected in the step 3 to process angle distortion caused by different lenses, door systems of different models, door opening modes of different types and the like, and the step 5 is carried out;
and 5: the detection recognition module loads a non-flat layer door opening classification model, judges whether the image processed in the step 4 is a non-flat layer door opening or not, and gives confidence
When is coming into contact with
If the threshold value is larger than the set threshold value, judging that the door is opened in a non-flat layer; otherwise, returning to the step 1; the confidence threshold of the present embodiment may be set to 90%.
The elevator non-flat layer fault recognition device and method based on computer vision provided by the embodiment of the invention can fully cover various types of elevator door systems, meet the requirement of rapid development of the elevator industry, realize intelligent recognition and prediction of opening doors of elevator non-flat layers, overcome the defects of the traditional detection mode, reduce the manual calibration workload of non-flat layer target samples, improve the target detection accuracy and recognition speed of elevator non-flat layer images and door opening and closing images, and achieve the model recognition accuracy of 95.3%. The embodiment adopts a leading elevator fault identification technology, and realizes extremely strong compatibility.
It should be noted that the above-described device embodiments are merely illustrative, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A computer vision-based elevator non-flat layer door opening fault recognition device is characterized by comprising a video acquisition module, an acceleration monitoring module, a model training module and a detection recognition module; wherein the content of the first and second substances,
the video acquisition module comprises a camera device and is used for acquiring an elevator sill image; the video acquisition module is electrically connected with the detection identification module; the lens of the camera device is arranged at a gap between the elevator car door and the hall door and is vertical to and right against the sill groove so as to shoot the hall door sill groove and the car door sill groove simultaneously;
the acceleration monitoring module comprises an acceleration sensor and a microprocessor; the acceleration monitoring module is used for monitoring the running acceleration of the elevator so as to pre-judge whether the elevator enters a deceleration and elevator stopping state, and the acceleration monitoring module is electrically connected with the detection and identification module;
the model training module comprises a sill slot target detection model training unit and a non-flat layer door opening classification model training unit; the sill trough target detection model training unit is used for pre-training a sill trough detection model for loading and using by the detection and identification module; the non-flat-layer door opening classification model training unit is used for pre-training a non-flat-layer door opening classification model for loading and using by the detection and recognition module;
and the detection and identification module is used for carrying out self-adaptive processing on an elevator sill image and loading a model trained by the model training module for prediction and identification when the elevator enters a deceleration and elevator stopping state according to the acceleration monitoring module so as to judge whether the elevator has a non-flat-layer door opening fault.
2. The elevator non-flat layer door opening fault recognition device based on the computer vision is characterized in that,
the microprocessor of the acceleration monitoring module is specifically used for detecting the acceleration of the acceleration curve after the filtering smoothing processing is finished by utilizing the variance, the range and the ratio of the index distance corresponding to the range to the length of the acceleration sequence so as to judge whether the elevator stops;
wherein the variance is used for measuring the fluctuation degree of the acceleration sequence, and the expression is the stability of the acceleration
Expressed as shown in equation (1):
wherein the content of the first and second substances,
indicating a sequence of accelerations
The value of each acceleration is taken as the value,
the acceleration mean value of the acceleration sequence is shown, n represents the length of the acceleration sequence, namely the acceleration sequence is a sequence consisting of n acceleration values;
the range is the maximum range for measuring the value variation, is the index for measuring the value variation, is the difference between the maximum value and the minimum value, and is used for measuring the value variation
Expressed as shown in equation (2):
wherein the content of the first and second substances,
represents the maximum value in the sequence of accelerations,
represents the minimum value of the acceleration sequence;
the acceleration is the shock degree which is described by the proportion of the index distance corresponding to the range difference to the length of the acceleration sequence, the proportion of the distance between the sequence positions of the maximum value and the minimum value to the whole acceleration sequence is described by
Expressed, as shown in equation (3):
wherein the content of the first and second substances,
indicating the sequence position in the acceleration sequence at which the maximum acceleration is located,
then representing the sequence position where the minimum acceleration is located, and the sequence position is represented as a few points; n is the length of the whole acceleration sequence;
when the variance is more than 1000, the range is more than 500 and/or the ratio of the index distance corresponding to the range to the length of the acceleration sequence is less than 10%, the elevator is considered to be about to stop; otherwise, the microprocessor judges that the acceleration is normal and the elevator is in normal operation.
3. The elevator non-flat layer door opening fault identification device based on the computer vision is characterized in that an acceleration sensor of the acceleration monitoring module has a plurality of acquisition time intervals; calculating the accuracy of the judgment of the microprocessor under each acquisition time interval by traversing the acquisition time interval of the acceleration sensor, and calculating the ratio of the accuracy to the information quantity acquired by the acceleration sensor in unit time; and when the ratio reaches the maximum value, setting the current corresponding acquisition time interval as a default acquisition time interval.
4. The elevator non-flat layer door opening fault recognition device based on the computer vision as claimed in claim 1, wherein the sill slot detection model training unit in the model training module comprises the following steps:
step 41: manufacturing a sill groove detection training set, and labeling a car picture containing a sill groove by using LabelImg, specifically framing out a sill groove image and labeling the type;
step 42: building a target detection model structure, using a model trained on an ILSVRC data set as an initial weight of a network backbone by adopting a transfer learning method, randomizing other parameters except the initial weight, and training on the sill-slot detection training set built in the step 41, wherein a loss function comprises a class classification loss function loss _ cls and a sill-slot boundary frame position return loss function loss _ bbox;
step 43: and setting a stopping strategy, and stopping training when the specified iteration times are reached or the prediction precision of the model in the verification set reaches a set value.
5. The elevator non-flat door opening fault recognition device based on the computer vision is characterized in that a non-flat door opening image classification model training unit in the model training module is used for training a non-flat door opening recognition model and performing two classifications on sill groove images to recognize a flat state;
the method for training the non-flat layer door opening classification model comprises the following steps:
step 51: the method comprises the steps of manufacturing a sill groove image classification training set, wherein in a door opening state, a sill groove image with the absolute value of the height difference between a hoistway door sill groove and a car door sill groove being less than or equal to 10mm is collected and used as a flat-layer door opening data set, and the rest sill groove image is a non-flat-layer door opening data set;
step 52: downloading and loading an inclusion V3 model as a pre-training model;
step 53: executing a transfer learning strategy, removing an output full connection layer and a softmax layer, adding a new two-classification full connection layer and a softmax classification layer, adding and deleting a preset intermediate network layer as an improved depth convolution neural network based on an initiation structure; the deep convolution neural network contains an inclusion structure, and different scale information of the image is extracted through multi-convolution kernel fusion to obtain better representation, wherein the inclusion structure contains a sparse network structure with 1 × 1 convolution to reduce dimension and build less parameters; the predetermined intermediate network layer comprises an initiation block, a 1 × 1 convolution;
step 54: loading the deep convolutional neural network obtained in the step 53, and creating a training script file;
step 55: preprocessing the sample of the classification training set of the sill-trough image, inputting the trained parameters of batch _ size, epoch and iteration, wherein the batch _ size represents the number of required samples used in one parameter updating process, 1 iteration is equal to 1 time of training by using the batch _ size samples, and the epoch is the total iteration times of all the samples;
step 56: setting a stopping strategy, and stopping training when iteration reaches a specified epoch number and iteration number or when the prediction precision of the model in the verification set reaches a set value;
and 57: a checkpoint mechanism is additionally arranged, and when the optimal value is reached in the process of training each epoch model, a model file is output;
step 58: and performing data preprocessing according to the batch _ size loading sample and then executing a training process.
6. The elevator non-flat layer door opening fault recognition device based on the computer vision is characterized in that training samples in a sill groove image classification training set are obtained by performing random cutting, overturning and feature transformation on original samples; the feature transformation is shown in the publications (4), (5) and (6):
wherein the content of the first and second substances,
representing a central pixel, for a given central point
The position of the circular neighborhood pixel is
And R is the radius of the sample,
is shown as
A number of sample points are sampled at the time of sampling,
is the number of samples to be taken,
is a sign function, if the difference between the adjacent position pixel and the central position pixel
Returns 1 when the number is positive or 0; otherwise, the value of 0 is returned to,
is a function of the rotation of the rotating body,
show that
Right circulation
A bit.
7. The elevator non-flat layer door opening fault recognition device based on the computer vision is characterized in that the detection and recognition module comprises an image illumination self-adaptive correction unit, a sill groove detection unit and an image angle self-adaptive correction unit; wherein
The image illumination self-adaptive correction unit is specifically used for performing illumination self-adaptive correction on an elevator sill image acquired by the video acquisition module and processing a sill groove image when the ambient illumination is too dark and too bright so as to improve the illumination uniformity and enhance the image quality, and the self-adaptive processing method is as shown in a formula (7) (8):
wherein，
Is the illumination characteristic of the corrected output image,
for input elevator sill image
Extracting illumination components, wherein m is a brightness mean value of the illumination components;
the coefficient is a self-adaptive coefficient, and the parameters k and r are variable parameters iteratively learned by a machine learning method according to the actual illumination condition of the elevator car;
the sill groove detection unit is used for: loading a target detection model to detect a sill slot, judging the opening degree alpha of the elevator according to the detected length of the sill slot and the ratio of the total length of the sill slot when the opening of the elevator is judged when the sill slot is detected, wherein the ratio of the detected actual length of the sill slot and the actual total length of the sill slot is equal to the corresponding pixel ratio of the detected actual length of the sill slot and the actual total length of the sill slot; when alpha is 0, the elevator is not opened; and when the door opening degree alpha is larger than or equal to the set threshold value, judging that the door is opened.
8. The elevator non-horizontal layer door opening fault recognition device based on computer vision of claim 7, wherein the image angle adaptive correction unit is specifically configured to perform angle transformation on the detected sill-slot image to adapt to angle distortion caused by different lenses, different door systems and different door opening modes, as shown in formulas (9) (10) (11) (12):
wherein the content of the first and second substances,
is the coordinates of the pixel points after the transformation,
is the coordinate of the pixel point before transformation,
respectively the width and the height of the image,
is the included angle between the center of the camera lens and the horizontal central line of the sill groove,
is the included angle between the center of the camera lens and the center line of the sill groove,
and
is formed by
And
and determining vectors for measuring the rotation angle of the two-dimensional plane and the rotation degree of the three-dimensional space around the z axis, and performing self-adaptive training by taking the vectors as the hyper-parameters.
9. The elevator non-flat layer door opening fault recognition device based on the computer vision is characterized in that a non-flat layer door opening recognition unit in the detection recognition module is specifically used for loading a non-flat layer door opening classification model, judging whether a hoistway door and a car door are not flat layers and giving confidence degree beta, and judging that the hoistway door and the car door are not flat layer doors when the beta is larger than a set threshold value.
10. A non-flat layer door opening fault identification method based on deep learning is characterized by comprising the following steps:
step 101: starting a video acquisition module; the acceleration monitoring module acquires an acceleration signal of the elevator car and transmits the acceleration signal to the microprocessor; the microprocessor receives the acceleration signal, carries out filtering smoothing processing, analyzes an acceleration curve after the filtering smoothing processing is finished, and enters step 102 if the acceleration curve is judged to enter a deceleration and elevator stopping stage, and continues to carry out acceleration monitoring if the acceleration curve is judged to normally run;
step 102: the detection and identification module receives the information of the acceleration monitoring module and starts to intercept an elevator car running video according to the monitored time for preparing to stop the elevator; the detection and identification module carries out frame processing on the intercepted video, carries out illumination self-adaptive correction on each frame of image and enters step 103;
step 103: loading a acamprosate groove detection model by a detection and identification module, carrying out target detection on the image obtained in the step 102, judging that the door is opened when the acamprosate groove is detected, and entering the step 104;
step 104: the detection and identification module performs angle self-adaptive correction on the sill-slot image detected in the step 103, intelligently processes image angle distortion caused by factors such as different lenses, door systems of different models, door opening modes of different types and the like, and enters a step 105;
step 105: the detection and recognition module loads a non-flat layer door opening classification model, judges whether the image processed in the step 104 is a non-flat layer door opening or not, gives a confidence coefficient beta, and judges that the image is a non-flat layer door opening when the beta is greater than a set threshold value; otherwise, go back to step 101.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110400660.9A CN112801072B (en) | 2021-04-14 | 2021-04-14 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110400660.9A CN112801072B (en) | 2021-04-14 | 2021-04-14 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112801072A true CN112801072A (en) | 2021-05-14 |
| CN112801072B CN112801072B (en) | 2021-11-02 |
Family
ID=75811344
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110400660.9A Active CN112801072B (en) | 2021-04-14 | 2021-04-14 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112801072B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114572787A (en) * | 2022-03-04 | 2022-06-03 | 陈小平 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011011872A (en) * | 2009-07-02 | 2011-01-20 | Mitsubishi Electric Building Techno Service Co Ltd | Failure detecting device for elevator |
| CN107416630A (en) * | 2017-09-05 | 2017-12-01 | 广州日滨科技发展有限公司 | The detection method and system of the improper closing of elevator |
| CN107539855A (en) * | 2017-08-07 | 2018-01-05 | 广州日滨科技发展有限公司 | The detection method and system of Elevator landing failure |
| CN107601199A (en) * | 2017-08-07 | 2018-01-19 | 广州日滨科技发展有限公司 | The detection method and system of elevator components failure |
| CN108341313A (en) * | 2018-03-28 | 2018-07-31 | 灏景（厦门）文化传媒股份有限公司 | A kind of method for early warning and its multimedia Forewarning Terminal of elevator safety operation environmental monitoring |
| CN108358004A (en) * | 2018-02-05 | 2018-08-03 | 苏州钜立智能系统有限公司 | Elevator operation monitoring method based on color identification technology |
| CN110127475A (en) * | 2019-03-27 | 2019-08-16 | 浙江新再灵科技股份有限公司 | A kind of method and system of elevator riding personnel classification and its boarding law-analysing |
| CN110407051A (en) * | 2019-08-02 | 2019-11-05 | 杭州岁丰信息技术有限公司 | A kind of identification lift running safety monitoring of audio-video and pacify system |
| CN111144247A (en) * | 2019-12-16 | 2020-05-12 | 浙江大学 | Escalator passenger reverse-running detection method based on deep learning |
| JP2020104983A (en) * | 2018-12-27 | 2020-07-09 | 株式会社日立製作所 | Device and method for elevator operation control |
| CN111731962A (en) * | 2020-06-29 | 2020-10-02 | 浙江新再灵科技股份有限公司 | Opening and closing fault detection method and detection early warning system for door |
| CN111792472A (en) * | 2020-06-03 | 2020-10-20 | 上海三菱电梯有限公司 | Control method and control system for elevator leveling position |
| CN112209193A (en) * | 2020-07-22 | 2021-01-12 | 上海城建职业学院 | Intelligent elevator system |
| CN112347862A (en) * | 2020-10-20 | 2021-02-09 | 浙江新再灵科技股份有限公司 | Elevator door fault real-time detection method based on machine vision |
| CN112357707A (en) * | 2020-10-21 | 2021-02-12 | 日立楼宇技术（广州）有限公司 | Elevator detection method and device, robot and storage medium |
| CN112591572A (en) * | 2020-12-03 | 2021-04-02 | 日立楼宇技术（广州）有限公司 | Elevator leveling precision verification system, method and device and controller |
-
2021
- 2021-04-14 CN CN202110400660.9A patent/CN112801072B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011011872A (en) * | 2009-07-02 | 2011-01-20 | Mitsubishi Electric Building Techno Service Co Ltd | Failure detecting device for elevator |
| CN107539855A (en) * | 2017-08-07 | 2018-01-05 | 广州日滨科技发展有限公司 | The detection method and system of Elevator landing failure |
| CN107601199A (en) * | 2017-08-07 | 2018-01-19 | 广州日滨科技发展有限公司 | The detection method and system of elevator components failure |
| CN107416630A (en) * | 2017-09-05 | 2017-12-01 | 广州日滨科技发展有限公司 | The detection method and system of the improper closing of elevator |
| CN108358004A (en) * | 2018-02-05 | 2018-08-03 | 苏州钜立智能系统有限公司 | Elevator operation monitoring method based on color identification technology |
| CN108341313A (en) * | 2018-03-28 | 2018-07-31 | 灏景（厦门）文化传媒股份有限公司 | A kind of method for early warning and its multimedia Forewarning Terminal of elevator safety operation environmental monitoring |
| JP2020104983A (en) * | 2018-12-27 | 2020-07-09 | 株式会社日立製作所 | Device and method for elevator operation control |
| CN110127475A (en) * | 2019-03-27 | 2019-08-16 | 浙江新再灵科技股份有限公司 | A kind of method and system of elevator riding personnel classification and its boarding law-analysing |
| CN110407051A (en) * | 2019-08-02 | 2019-11-05 | 杭州岁丰信息技术有限公司 | A kind of identification lift running safety monitoring of audio-video and pacify system |
| CN111144247A (en) * | 2019-12-16 | 2020-05-12 | 浙江大学 | Escalator passenger reverse-running detection method based on deep learning |
| CN111792472A (en) * | 2020-06-03 | 2020-10-20 | 上海三菱电梯有限公司 | Control method and control system for elevator leveling position |
| CN111731962A (en) * | 2020-06-29 | 2020-10-02 | 浙江新再灵科技股份有限公司 | Opening and closing fault detection method and detection early warning system for door |
| CN112209193A (en) * | 2020-07-22 | 2021-01-12 | 上海城建职业学院 | Intelligent elevator system |
| CN112347862A (en) * | 2020-10-20 | 2021-02-09 | 浙江新再灵科技股份有限公司 | Elevator door fault real-time detection method based on machine vision |
| CN112357707A (en) * | 2020-10-21 | 2021-02-12 | 日立楼宇技术（广州）有限公司 | Elevator detection method and device, robot and storage medium |
| CN112591572A (en) * | 2020-12-03 | 2021-04-02 | 日立楼宇技术（广州）有限公司 | Elevator leveling precision verification system, method and device and controller |
Non-Patent Citations (5)
| Title |
|---|
| CHIH-YU HSU 等: "Machine Learning Modeling for Failure Detection of Elevator Doors by Three-Dimensional Video Monitoring", 《IEEE ACCESS》 * |
| PENGGAO WEN 等: "Fault Prediction of Elevator Door System Based on PSO-BP Neural Network", 《ENGINEERING》 * |
| 张霍: "基于图像与加速度传感器的电梯开关门检测", 《电子世界》 * |
| 胡欢 等: "基于计算机视觉算法的电梯故障预测系统", 《单片机与嵌入式系统应用》 * |
| 赵海文 等: "基于YOLO模型的机器人电梯厅门装箱状态快速识别方法", 《包装工程》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114572787A (en) * | 2022-03-04 | 2022-06-03 | 陈小平 | Elevator non-flat-layer door opening fault recognition device and method based on computer vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112801072B (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111681240B (en) | Bridge surface crack detection method based on YOLO v3 and attention mechanism | |
| CN111784633B (en) | Insulator defect automatic detection algorithm for electric power inspection video | |
| CN110781896B (en) | Track garbage identification method, cleaning method, system and resource allocation method | |
| CN111797890A (en) | Method and system for detecting defects of power transmission line equipment | |
| CN107220603A (en) | Vehicle checking method and device based on deep learning | |
| CN113205063A (en) | Visual identification and positioning method for defects of power transmission conductor | |
| CN109935080B (en) | Monitoring system and method for real-time calculation of traffic flow on traffic line | |
| CN107081765A (en) | A kind of substation inspection robot autonomous classification method and a kind of inspecting robot | |
| CN109711322A (en) | A kind of people's vehicle separation method based on RFCN | |
| CN112528979B (en) | Transformer substation inspection robot obstacle distinguishing method and system | |
| CN111862065A (en) | Power transmission line diagnosis method and system based on multitask deep convolutional neural network | |
| CN113313107A (en) | Intelligent detection and identification method for multiple types of diseases on cable surface of cable-stayed bridge | |
| CN112801072B (en) | Elevator non-flat-layer door opening fault recognition device and method based on computer vision | |
| CN115841633A (en) | Power tower and power line associated correction power tower and power line detection method | |
| CN109086781A (en) | A kind of cabinet lamp state identification method based on deep learning | |
| CN112115770A (en) | Method and system for identifying autonomous inspection defects of unmanned aerial vehicle of overhead line | |
| CN111950357A (en) | Marine water surface garbage rapid identification method based on multi-feature YOLOV3 | |
| CN113065454B (en) | High-altitude parabolic target identification and comparison method and device | |
| CN108563986A (en) | Earthquake region electric pole posture judgment method based on wide-long shot image and system | |
| CN112508911A (en) | Rail joint touch net suspension support component crack detection system based on inspection robot and detection method thereof | |
| CN112801071B (en) | Elevator asynchronous door opening recognition system and method based on deep learning | |
| CN111667655A (en) | Infrared image-based high-speed railway safety area intrusion alarm device and method | |
| CN110163081A (en) | Regional invasion real-time detection method, system and storage medium based on SSD | |
| CN115931895A (en) | Method and device for checking defects of power transmission line | |
| CN109359518A (en) | A kind of moving object recognition methods, system and the warning device of infrared video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |