CN113435466A - Method, device, medium and terminal for detecting elevator door position and switch state - Google Patents
Method, device, medium and terminal for detecting elevator door position and switch state Download PDFInfo
- Publication number
- CN113435466A CN113435466A CN202011570412.0A CN202011570412A CN113435466A CN 113435466 A CN113435466 A CN 113435466A CN 202011570412 A CN202011570412 A CN 202011570412A CN 113435466 A CN113435466 A CN 113435466A
- Authority
- CN
- China
- Prior art keywords
- elevator door
- network model
- elevator
- detection
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000001514 detection method Methods 0.000 claims abstract description 126
- 238000012549 training Methods 0.000 claims abstract description 61
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 18
- 238000004364 calculation method Methods 0.000 claims abstract description 10
- 238000010586 diagram Methods 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004927 fusion Effects 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000011478 gradient descent method Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 230000015654 memory Effects 0.000 abstract description 18
- 230000008569 process Effects 0.000 abstract description 7
- 238000012805 post-processing Methods 0.000 abstract description 4
- 238000013528 artificial neural network Methods 0.000 abstract description 3
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B66—HOISTING; LIFTING; HAULING
- B66B—ELEVATORS; ESCALATORS OR MOVING WALKWAYS
- B66B13/00—Doors, gates, or other apparatus controlling access to, or exit from, cages or lift well landings
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Indicating And Signalling Devices For Elevators (AREA)
Abstract
The invention discloses a method, a device, a medium and a terminal for detecting the position and the opening and closing state of an elevator door, wherein the method comprises the following steps: training an elevator door detection network model based on a first-order full convolution target detection algorithm; and acquiring a real-time image of the elevator door, and identifying the position and the opening and closing state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model. The method is based on a first-order full-convolution target detection algorithm (FCOS), a priori frame is not required to be set in the neural network training and detecting process, the number of parameters is obviously reduced, massive calculation is avoided, memory occupation is reduced, and the training and detecting process is simple and efficient; meanwhile, after the detection is finished, a weighted post-processing method is adopted for confidence calculation, so that the number of redundant frames is reduced, and the accuracy of the detection of the elevator door position and the opening and closing state is improved.
Description
Technical Field
The invention relates to the field of robots, in particular to a method, a device, a medium and a terminal for detecting elevator door positions and opening and closing states.
Background
The traditional technology generally adopts a Hall sensor to judge the state of the elevator door, so equipment or the sensor needs to be additionally arranged on the elevator door, the mode is easily interfered by electromagnetic interference and complex in installation, and once the position is deviated, the state of the elevator door is easily judged wrongly, so that the fault of the elevator door is mistakenly reported and is not reported, and meanwhile, the fault rate of the sensor also influences the state judgment of the elevator door. With the development of artificial intelligence technology, the computer vision technology based on deep learning can be used for analyzing and judging the state of the elevator door. Generally speaking, the state of the elevator door can be directly identified by using the objective detection technology such as SSD, YOLO, etc., but the size, the aspect ratio, the number, etc. of the prior frame need to be set reasonably first, which not only has many parameters and troublesome setting, but also needs to set a large number of prior frames to achieve a high recall rate, and has a large calculation amount and a high memory occupation.
Disclosure of Invention
The invention provides a method, a device, a medium and a terminal for detecting the position and the opening and closing state of an elevator door, and solves the technical problems of complex steps, multiple parameters and large calculated amount caused by the fact that a prior frame needs to be set in the prior art for target detection.
The technical scheme for solving the technical problems is as follows: a method for detecting the position and the opening and closing state of an elevator door comprises the following steps:
training an elevator door detection network model based on a first-order full convolution target detection algorithm;
and acquiring a real-time image of the elevator door, and identifying the position of the elevator door and the opening and closing state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model.
In a preferred embodiment, the training of the elevator door detection network model based on the first-order full convolution target detection algorithm specifically includes:
acquiring elevator door images of each elevator hall at random and multiple angles through a camera of the mobile robot, marking the elevator door position and the opening and closing state of the elevator door in the elevator door images, and establishing a training set;
constructing an elevator door detection network model based on a first-order full convolution target detection algorithm, wherein the elevator door detection network model comprises a backbone network, a characteristic pyramid network and a full convolution detection head module which are sequentially connected, and the full convolution detection head module comprises a classification branch, a centrality branch and a square frame regression branch;
initializing network parameters of the elevator door detection network model to generate an initial weight and an initial bias;
inputting all images of the training set into an initialized elevator door detection network model, extracting a feature map of an input image through the backbone network, and performing multi-scale feature fusion on the feature map through the feature pyramid network to generate feature maps with different scales;
generating a size prediction value of each feature point returning elevator door frame on the feature maps with different scales through the frame regression branch, generating a nearness prediction value of each feature point from the center position of the elevator door frame through the center degree branch, generating an elevator door opening and closing state prediction value of each feature point through the classification branch, and calculating a loss value according to a preset loss function;
and reducing the loss value and performing backward propagation, and updating the weight and the bias of the elevator door detection network model through repeated circulating forward propagation and backward propagation until a preset iteration stop condition is reached to generate the trained elevator door detection network model.
In a preferred embodiment, the predetermined loss function is:
Loss＝LossBbox+LossCls+LossCenter，
wherein, lossbox ═ GIOULoss (Bbox)pred，Bboxgt)，
union＝areaA+areaB-areain，
areain＝(inl+inr)*(int+ind)，
areaout＝(outl+outr)*(outt+outd)，
inl＝min(al，bl)，inr＝min(ar，br)，
int＝min(at，bt)，ind＝min(ad，bd)，
outl＝max(al，bl)，outr＝max(ar，br)，
outt＝max(at，bt)，outd＝max(ad，bd)，
LossCls＝Clsgt*log(Clspred)+(1-Clsgt)*log(1-Clspred)，
LossCenter＝Ctgt*log(Ctpred)+(1-Ctgt)*log(1-Ctpred)，
BboxpredRepresenting the predicted value of the size of each feature point regression elevator door square frame, BboxgtRepresents the training target value, Ct, of each feature point regression elevator door square framepredRepresenting the predicted value of the distance between each characteristic point and the center of the elevator door frame, CtgtTraining target value of distance, Cls, representing the position of each feature point from the center of elevator door framepredElevator door open/close state prediction value, Cls, representing each feature pointgtRepresenting the elevator door opening and closing state training target value of each characteristic point; al represents the upper left abscissa of box a, bl represents the upper left abscissa of box b, ar represents the lower right abscissa of box a, br represents the lower right abscissa of box b, at represents the upper left ordinate of box a, ad represents the lower right ordinate of box a, bt represents the upper left ordinate of box b, and bd represents the lower right ordinate of box b.
In a preferred embodiment, the preset loss function is subjected to minimum calculation by adopting a driving quantity random gradient descent method with a momentum parameter of 0.9, weight attenuation of 0.001 and slow descent of a learning rate polynomial, the training is terminated after 100 times of training, the network parameters of the elevator door detection network model are stored, and the elevator door detection network model after the training is generated.
In a preferred embodiment, the acquiring a real-time image of an elevator door, and identifying the position and the open/close state of the elevator door in the real-time image of the elevator door based on a trained elevator door detection network model specifically includes:
randomly acquiring a real-time image of an elevator door of an elevator hall through a camera of the mobile robot;
performing elevator door position detection and elevator door opening and closing state detection on the elevator door real-time image by adopting a trained elevator door detection network model;
and outputting the elevator door square frame which reaches a preset confidence threshold and has the overlapping rate less than or equal to 5%.
In a preferred embodiment, the preset confidence threshold is 0.5, and the confidence formula for calculating the elevator door box is as follows:
obj＝Clspred 0.3*Ctpred 0.7，
wherein, ClspredThe predicted value, Ct, of the opening and closing state of the elevator door representing each characteristic pointpredAnd (4) showing a predicted value of the distance between each characteristic point and the center position of the elevator door frame.
A second aspect of an embodiment of the present invention provides a device for detecting an elevator door position and an open/close state, including a training module and a detection module,
the training module is used for training an elevator door detection network model based on a first-order full convolution target detection algorithm;
the detection module is used for acquiring a real-time image of the elevator door, and identifying the position of the elevator door and the opening and closing state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model.
In a preferred embodiment, the training module comprises:
the system comprises a preprocessing unit, a data processing unit and a data processing unit, wherein the preprocessing unit is used for acquiring elevator door images of each elevator hall at random and multiple angles through a camera of a mobile robot, marking the positions of the elevator doors and the opening and closing states of the elevator doors in the elevator door images and establishing a training set;
the elevator door detection network model comprises a backbone network, a characteristic golden tower network and a full convolution detection head module which are sequentially connected, wherein the full convolution detection head module comprises a classification branch, a centrality branch and a square frame regression branch;
the initialization unit is used for initializing the network parameters of the elevator door detection network model to generate an initial weight and an initial bias;
the characteristic extraction unit is used for inputting all images of the training set into the initialized elevator door detection network model, extracting a characteristic diagram of the input image through the backbone network, and performing multi-scale characteristic fusion on the characteristic diagram through the characteristic pyramid network to generate characteristic diagrams with different scales;
the training unit is used for generating a size predicted value of each feature point on the feature map with different scales for regressing the elevator door frame through the frame regression branch, generating a nearness predicted value of each feature point from the center position of the elevator door frame through the center degree branch, generating an elevator door opening and closing state predicted value of each feature point through the classification branch, and calculating a loss value according to a preset loss function; and the weight and the bias of the elevator door detection network model are updated through repeated circulating forward propagation and backward propagation until a preset iteration stop condition is reached, and the trained elevator door detection network model is generated.
A third aspect of embodiments of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of the above-described method for detecting an elevator door position and a switch state.
A fourth aspect of the present invention provides a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the above method for detecting elevator door position and switch state when executing the computer program.
The invention provides a method, a device, a medium and a terminal for detecting the position and the on-off state of an elevator door, which are based on a first-order full-convolution target detection algorithm (FCOS), do not need to set a prior frame in the training and detection process of a neural network, not only obviously reduce the number of parameters, but also avoid a large amount of calculation, reduce the memory occupation, and have simple and efficient training and detection processes; meanwhile, after the detection is finished, a weighted post-processing method is adopted for confidence calculation, so that the number of redundant frames is reduced, and the accuracy of the detection of the elevator door position and the opening and closing state is improved.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments are briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention, and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to these drawings without inventive efforts.
Fig. 1 is a flow chart showing a method for detecting an elevator door position and an open/close state provided in embodiment 1;
fig. 2 is a block diagram of a backbone network of an elevator door detection network model;
fig. 3 is a schematic structural view of a full convolution detecting head module of an elevator door detecting network model;
FIG. 4 is a schematic diagram of a size prediction value of an elevator door frame of the regression of each characteristic point by an elevator door detection network model positive plate;
fig. 5 is a schematic diagram showing the structure of the device for detecting the door position and the open/close state of an elevator provided in embodiment 2;
fig. 6 is a schematic circuit diagram of a controller provided in embodiment 3.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without any inventive step, are within the scope of the present invention.
It should be noted that, if not conflicted, the various features of the embodiments of the invention may be combined with each other within the scope of protection of the invention. Additionally, while functional block divisions are performed in device schematics, with logical sequences shown in flowcharts, in some cases, the steps shown or described may be performed in a different order than the block divisions in the devices, or in the flowcharts. The terms "first", "second", "third", and the like used in the present invention do not limit data and execution order, but distinguish the same items or similar items having substantially the same function and action.
Referring to fig. 1, a schematic flow chart of a method for detecting an elevator door position and an open-close state is provided for embodiment 1 of the present invention, and as shown in fig. 1, the method includes the following steps:
step 1, training an elevator door detection network model based on a first-order full convolution target detection algorithm.
A first-order full-convolution Object Detection algorithm (FCOS) is a pixel-by-pixel Object Detection method based on a full-convolution network. The target detection method is used for extracting features of an input picture and carrying out target classification and bounding box regression by taking each pixel point in a feature map as a center. In a preferred embodiment, the training of the elevator door detection network model based on the first-order full convolution target detection algorithm specifically includes the following steps:
s101, acquiring elevator door images of each elevator hall at random and multiple angles through a camera of the mobile robot, marking the elevator door position and the opening and closing state of the elevator door in the elevator door images, and establishing a training set. Because the opening and closing state of the elevator door is complex, when the elevator door image is collected, the number ratio of the elevator door opening to the elevator door opening is kept to be 2: 1, meanwhile, the elevator door in the collected elevator door image is marked, the label is divided into opening and closing, and the states of virtual covering, half opening and the like of the elevator door are unified and merged into the elevator door opening.
S102, then constructing an elevator door detection network model based on a first-order full convolution target detection algorithm.
The elevator door has regular target size which is usually rectangular, and the invention combines the scene characteristics, selects a first-order full-volume target detection algorithm (FCOS), and defines all points [ x, y ] on the characteristic diagram as positive samples when falling into the elevator door frame, thereby improving the recall rate of elevator door detection.
Specifically, the constructed elevator door detection network model comprises a backbone network, a characteristic pyramid network and a full convolution detection head module which are sequentially connected, wherein the full convolution detection head module comprises a classification branch, a centrality branch and a square regression branch. Fig. 2 is a structural diagram of the backbone network in the preferred embodiment, and as shown in fig. 2, the network structure of the backbone network includes, in order of computing units:
l1, first convolution unit, input size 320 × 3, convolution layer 32 channels, 3 × 3 convolution kernel, step size 2; connecting BN layer processing and then connecting an LeakyReLU layer output;
l2, second convolution unit, input size 160 × 32, convolution layer 64 channels, 3 × 3 convolution kernel, step stride 2; connecting BN layer processing and then connecting an LeakyReLU layer output;
l3, third convolution unit, input size 80 x 64, convolution layer 128 channels, 3x3 convolution kernel, step stride 2; connecting BN layer processing and then connecting an LeakyReLU layer output;
l4, fourth convolution unit, input size 40 × 128, convolution layer: 128 channels, 3 × 3 convolution kernel, step size stride of 2; connecting BN layer processing and then connecting an LeakyReLU layer output;
l5, fifth convolution unit, input size 20 × 128, convolution layer 256 channels, 3 × 3 convolution kernel, step stride 2; and (4) connecting BN layer processing, and then connecting a LeakyReLU layer, and outputting the output size of 10 × 256.
Fig. 3 is a schematic structural diagram of the full convolution detecting head module in a preferred embodiment, and as shown in fig. 3, the full convolution detecting head module includes:
l6, sixth convolution unit, input size 10 × 256, convolution layer 128 channels, 1 × 1 convolution kernel, step stride 1; connecting BN layer processing and then connecting an LeakyReLU layer output;
detecting the left branch of the head module: a seventh convolution unit with input size 10 × 128, convolution layer 128 channels, 3 × 3 convolution kernel, step size stride 1; connecting BN layer processing and then connecting an LeakyReLU layer output;
detecting the left classification branch of the head module: a first convolution layer with an input size of 10 x 128, 128 channels, 1x1 convolution kernel, step stride of 1; outputting a characteristic diagram of 10 x 2;
detecting the left central branch of the head module: a second convolutional layer with input size of 10 × 128, 128 channels, 1 × 1 convolution kernel, step size stride of 1; outputting a characteristic diagram of 10 × 1;
detecting the right branch of the head module: an eighth convolution unit with an input size of 10 × 128, a convolution layer of 128 channels, a convolution kernel of 3 × 3, and a step stride of 1; connecting BN layer processing and then connecting an LeakyReLU layer output;
detecting the right square frame regression branch of the head module: and the third convolution layer has the input size of 10 × 128, the convolution layer has 128 channels, the convolution kernel has the size of 1 × 1, the step size stride is 1, and the output is a characteristic diagram of 10 × 4.
Then, S103 is executed to initialize the network parameters of the elevator door detection network model, and generate an initial weight and an initial bias. In a specific embodiment, the initialization may be performed using a lmagNet pre-training weight.
And S104, inputting all the images of the training set into the initialized elevator door detection network model, extracting the feature map of the input image through the backbone network, and performing multi-scale feature fusion on the feature map through the feature pyramid network to generate feature maps with different scales.
And S105, generating a size prediction value of each feature point on the feature map with different scales regressing the elevator door frame through the frame regression branch. Specifically, the function of the block regression branch is regression size, and for any point in the marked elevator door block, the distance from the four sides of the elevator door block is the target size to be regressed by the feature point, so that each feature point on the feature map should regress by 4 distances, i.e., d _ left, d _ right, d _ top, and d _ down, as shown in fig. 4. The target of the block regression branch is bbox _ gt H//32, W//32, 4, H, W is the picture size, after the picture is calculated by the network, the size of the output feature map is reduced to 32 times of the original size, thus H//32, W//32, and the block needs to predict four parameters x1, y1, x2, y2, thus the last value is 4.
And meanwhile, generating a predicted value of the distance between each characteristic point and the central position of the elevator door frame through the central degree branch. The centrality branch is defined here as:
namely, each feature point corresponds to a centrality, and the training target of the centrality branch is Ct _ gt [ H//32, W//32, 1 ]. H. W is the picture size, after the picture is calculated by the network, the size of the output feature map is reduced to 32 times of the original size, so H//32, W//32, and the central branch needs to predict the confidence of the gate at the corresponding position of the feature map, so the last value is 1.
It is also necessary to generate an elevator door open/close state prediction value for each feature point through the classification branch. Specifically, the classification branch predicts two characteristic graphs [ H//32, W//32, 2] simultaneously when processing elevator door classification, wherein the [ H//32, W//32, 0] characteristic graph predicts the closed elevator door, namely 0 represents the confidence of the door closure; [ H//32, W//32, 1] signature predicts an open elevator door, i.e. 1 represents a confidence level of the door opening. For example, when the elevator door is opened, all areas marked by the elevator door frame of the elevator door correspond to the position training targets on the characteristic diagram [ H//32, W//32, 1] as positive samples, namely 1, the rest positions are negative samples, namely assigned with 0, and the target of the training classification branch is cls _ gt [ H//32, W//32, 2 ].
Then, calculating a loss value according to a preset loss function, wherein the preset loss function is as follows:
Loss＝LossBbox+LossCls+LossCenter，
wherein, lossbox ═ GIOULoss (Bbox)pred，Bboxgt)，
union＝areaA+areaB-areain，
areain＝(inl+inr)*(int+ind)，
areaout＝(outl+outr)*(outt+outd)，
inl＝min(al，bl)，inr＝min(ar，br)，
int＝min(at，bt)，ind＝min(ad，bd)，
outl＝max(al，bl)，outr＝max(ar，br)，
outt＝max(at，bt)，outd＝max(ad，bd)，
LossCls＝Clsgt*log(Clspred)+(1-Clsgt)*log(1-Clspred)，
LossCenter＝Ctgt*log(Ctpred)+(1-Ctgt)*log(1-Ctpred)，
BboxpredAnd (3) representing the size prediction value of each feature point regression elevator door frame, wherein the elevator door frame is represented by coordinates (x1, y1, x2 and y2), wherein x1 and y1 are coordinates of the upper left corner of the frame, and x2 and y2 are coordinates of the lower right corner of the frame. BboxgtRepresents the training target value, Ct, of each feature point regression elevator door squarepredRepresenting the distance prediction value, Ct, of each feature point from the center of the elevator door framegtTraining target value of distance, Cls, of each feature point from center of elevator door framepredElevator door open/close state prediction value, Cls, representing each feature pointgtRepresent each oneAnd training the opening and closing state of the elevator door at the characteristic point to obtain a target value. area A represents the area of box a, area B represents the area of box b, al represents the upper left abscissa of box a, bl represents the upper left abscissa of box b, ar represents the lower right abscissa of box a, br represents the lower right abscissa of box b, at represents the upper left ordinate of box a, ad represents the lower right ordinate of box a, bt represents the upper left ordinate of box b, and bd represents the lower right ordinate of box b.
And S106, reducing the loss value and performing back propagation in the training process, and updating the weight and the bias of the elevator door detection network model through repeated cycle forward propagation and back propagation until a preset iteration stop condition is reached to generate the trained elevator door detection network model. In a preferred embodiment, the preset loss function is subjected to minimum calculation by adopting a driving quantity random gradient descent method, training is terminated after 100 times of training, network parameters of the elevator door detection network model are stored, and the trained elevator door detection network model is generated. The momentum parameter of the stochastic gradient descent method with momentum is 0.9, the weight attenuation is 0.001, and the learning rate polynomial slowly descends.
And then step 2 is executed, elevator door real-time images are collected, and the position and the opening and closing state of the elevator door in the elevator door real-time images are identified based on the trained elevator door detection network model. The method specifically comprises the following steps:
s201, randomly acquiring a real-time image of an elevator door of an elevator hall through a camera of the mobile robot;
s202, performing elevator door position detection and elevator door opening and closing state detection on the elevator door real-time image by adopting the trained elevator door detection network model;
and S203, outputting the elevator door square frame which reaches the preset confidence level threshold value and has the overlapping rate less than or equal to 5%.
In the scene related to the invention, the elevator doors are distributed dispersedly, and only one elevator door is arranged in one area, so that the elevator doors can not be overlapped in the actual scene. Therefore, to reduce the number of redundant boxes, a weighted post-processing (weighted post-process) method is proposed, with the door confidence defined as follows:
obj＝Clspred 0.3*Ctpred 0.7，
wherein, ClspredThe predicted value, Ct, of the opening and closing state of the elevator door representing each characteristic pointpredAnd (4) showing a predicted value of the distance between each characteristic point and the center position of the elevator door frame. The elevator door confidence level is neglected to be less than 0.5, and the redundant box with the overlapping rate of more than 5 percent is deleted.
The invention provides a method for detecting the position and the on-off state of an elevator door, which is based on a first-order complete convolution target detection algorithm (FCOS), and does not need to set a first-check frame in the training and detecting process of a neural network, thereby not only obviously reducing the number of parameters, but also avoiding a large amount of calculation, reducing the memory occupation, and having simple and efficient training and detecting process; meanwhile, after detection is finished, confidence coefficient calculation is carried out by adopting a weighted post-processing method, so that the number of redundant frames is reduced, and the accuracy of detection of the position and the opening and closing state of the elevator door is improved.
It should be noted that, in the foregoing embodiments, a certain order does not necessarily exist between the steps, and it can be understood by those skilled in the art from the description of the embodiments of the present invention that, in different embodiments, the steps may have different execution orders, that is, may be executed in parallel, may also be executed interchangeably, and the like.
As another aspect of the embodiment of the present invention, the embodiment of the present invention further provides a device for detecting a position and an open/close state of an elevator door. The device for detecting the elevator door position and the switch state can be a software module, the software module comprises a plurality of instructions which are stored in a memory, and a processor can access the memory and call the instructions to execute the instructions so as to complete the method for detecting the elevator door position and the switch state explained in the embodiments.
In some embodiments, the device for detecting the elevator door position and the opening/closing state may also be built by hardware devices, for example, the device for detecting the elevator door position and the opening/closing state may be built by one or more than two chips, and each chip may work in coordination with each other to complete the method for detecting the elevator door position and the opening/closing state described in each of the above embodiments. For another example, the detection device for the position and the opening/closing state of the elevator door may be constructed by various logic devices, such as a general processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a single chip microcomputer, an arm (aconris cmachine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.
Fig. 5 is a schematic structural diagram of an elevator door position and open-close state detection device provided in embodiment 2 of the present invention, which includes a training module 100 and a detection module 200,
the training module 100 is used for training an elevator door detection network model based on a first-order full convolution target detection algorithm;
the detection module 200 is configured to acquire a real-time image of an elevator door, and identify a position of the elevator door and an open/close state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model.
In a preferred embodiment, the training module 100 comprises:
the system comprises a preprocessing unit, a data processing unit and a data processing unit, wherein the preprocessing unit is used for acquiring elevator door images of each elevator hall at random and multiple angles through a camera of a mobile robot, marking the positions of the elevator doors and the opening and closing states of the elevator doors in the elevator door images and establishing a training set;
the elevator door detection network model comprises a backbone network, a characteristic golden tower network and a full convolution detection head module which are sequentially connected, wherein the full convolution detection head module comprises a classification branch, a centrality branch and a square frame regression branch;
the initialization unit is used for initializing the network parameters of the elevator door detection network model to generate an initial weight and an initial bias;
the characteristic extraction unit is used for inputting all images of the training set into the initialized elevator door detection network model, extracting a characteristic diagram of the input image through the backbone network, and performing multi-scale characteristic fusion on the characteristic diagram through the characteristic pyramid network to generate characteristic diagrams with different scales;
the training unit is used for generating a size predicted value of each feature point on the feature map with different scales for regressing the elevator door frame through the frame regression branch, generating a nearness predicted value of each feature point from the center position of the elevator door frame through the center degree branch, generating an elevator door opening and closing state predicted value of each feature point through the classification branch, and calculating a loss value according to a preset loss function; and the weight and the bias of the elevator door detection network model are updated through repeated circulating forward propagation and backward propagation until a preset iteration stop condition is reached, and the trained elevator door detection network model is generated.
The elevator door position and opening/closing state detection device can execute the elevator door position and opening/closing state detection method provided by the embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details which are not described in detail in the embodiment of the device for detecting the elevator door position and the opening and closing state, reference is made to the method for detecting the elevator door position and the opening and closing state provided by the embodiment of the invention.
Fig. 6 is a schematic circuit diagram of a controller according to an embodiment of the present invention. As shown in fig. 3, the controller 600 includes one or more processors 61 and a memory 62. In fig. 3, one processor 61 is taken as an example.
The processor 61 and the memory 62 may be connected by a bus or other means, such as the bus connection in fig. 3.
The memory 62, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for detecting elevator door position and switch state in embodiments of the present invention. The processor 61 executes various functional applications and data processing of the elevator door position and open-close state detection device by running the nonvolatile software program, instructions and modules stored in the memory 62, that is, the elevator door position and open-close state detection method provided by the above method embodiment and the functions of the various modules or units of the above device embodiment are realized.
The memory 62 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state memory device. In some embodiments, the memory 62 may optionally include memory located remotely from the processor 61, and these remote memories may be connected to the processor 61 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The program instructions/modules are stored in the memory 62 and, when executed by the one or more processors 61, perform the method of detecting elevator door position and switch state in any of the method embodiments described above.
Embodiments of the present invention also provide a non-transitory computer storage medium storing computer-executable instructions for execution by one or more processors, such as a processor 61 in fig. 6, to enable the one or more processors to perform the method for detecting elevator door position and switch state in any of the above method embodiments.
Embodiments of the present invention also provide a computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions which, when executed by an electronic device, cause the electronic device to perform any of the elevator door position and switch state detection methods.
The above-described embodiments of the apparatus or device are merely illustrative, wherein the unit modules described as separate parts may or may not be physically separate, and the parts displayed as module units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a general hardware platform, and certainly can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or contributing to the related art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; within the idea of the invention, also technical features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for detecting the position and the opening and closing state of an elevator door is characterized by comprising the following steps:
training an elevator door detection network model based on a first-order full convolution target detection algorithm;
and acquiring a real-time image of the elevator door, and identifying the position and the opening and closing state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model.
2. The method for detecting the door position and the open-close state of an elevator according to claim 1, characterized in that the training of the elevator door detection network model based on the first-order full convolution target detection algorithm specifically comprises the following steps:
acquiring elevator door images of each elevator hall at random and multiple angles through a camera of the mobile robot, marking the elevator door position and the opening and closing state of the elevator door in the elevator door images, and establishing a training set;
constructing an elevator door detection network model based on a first-order full convolution target detection algorithm, wherein the elevator door detection network model comprises a backbone network, a characteristic pyramid network and a full convolution detection head module which are sequentially connected, and the full convolution detection head module comprises a classification branch, a centrality branch and a square frame regression branch;
initializing network parameters of the elevator door detection network model to generate an initial weight and an initial bias;
inputting all images of the training set into an initialized elevator door detection network model, extracting a characteristic diagram of an input image through the backbone network, and performing multi-scale characteristic fusion on the characteristic diagram through the characteristic pyramid network to generate different-scale characteristic diagrams;
generating a size prediction value of each feature point regression elevator door frame on the feature map with different scales through the frame regression branch, generating a nearness prediction value of each feature point from the center position of the elevator door frame through the center degree branch, generating an elevator door opening and closing state prediction value of each feature point through the classification branch, and then calculating a loss value according to a preset loss function;
and reducing the loss value and performing back propagation, and updating the weight and the bias of the elevator door detection network model through repeated cycle forward propagation and back propagation until a preset iteration stop condition is reached to generate the trained elevator door detection network model.
3. The method for detecting elevator door position and switch status according to claim 2 wherein the predetermined loss function is:
Loss＝LossBbox+LossCls+LossCenter，
wherein, lossbox ═ GIOULoss (Bbox)pred，Bboxgt)，
union＝areaA+areaB-areain，
areain＝(inl+inr)*(int+ind)，
areaout＝(outl+outr)*(outt+outd)，
inl＝min(al，bl)，inr＝min(ar，br)，
int＝min(at，bt)，ind＝min(ad，bd)，
outl＝max(al，bl)，outr＝max(ar，br)，
outt＝max(at，bt)，outd＝max(ad，bd)，
LossCls＝Clsgt*log(Clspred)+(1-Clsgt)*log(1-Clspred)，
LossCenter＝Ctgt*log(Ctpred)+(1-Ctgt)*log(1-Ctpred)，
BboxpredRepresenting the predicted value of the size of each feature point regression elevator door square frame, BboxgtRepresents the training target value, Ct, of each feature point regression elevator door squarepredRepresenting the predicted value of the distance between each characteristic point and the center of the elevator door frame, CtgtTraining target value of distance, Cls, representing the position of each feature point from the center of elevator door framepredElevator door open/close state prediction value, Cls, representing each feature pointgtRepresenting the elevator door opening and closing state training target value of each characteristic point; al represents the squareThe abscissa of the upper left corner of the box a, bl represents the abscissa of the upper left corner of the box b, ar represents the abscissa of the lower right corner of the box a, br represents the abscissa of the lower right corner of the box b, at represents the ordinate of the upper left corner of the box a, ad represents the ordinate of the lower right corner of the box a, bt represents the ordinate of the upper left corner of the box b, and bd represents the ordinate of the lower right corner of the box b.
4. The method for detecting the door position and the open-close state of an elevator according to claim 3, characterized in that the driving quantity random gradient descent method is adopted to carry out the minimum calculation on the preset loss function, the training is terminated after 100 times of training, the network parameters of the elevator door detection network model are saved, and the trained elevator door detection network model is generated; the momentum parameter of the driving quantity random gradient descent method is 0.9, the weight attenuation is 0.001, and the learning rate polynomial descends slowly.
5. The method for detecting the door position and the open-close state of the elevator according to any one of claims 1 to 4, wherein the step of acquiring a real-time image of the elevator door and identifying the door position and the open-close state of the elevator door in the real-time image of the elevator door based on a trained elevator door detection network model specifically comprises the steps of:
randomly acquiring a real-time image of an elevator door of an elevator hall through a camera of the mobile robot;
performing elevator door position detection and elevator door opening and closing state detection on the elevator door real-time image by adopting a trained elevator door detection network model;
and outputting the elevator door square frame which reaches a preset confidence threshold and has the overlapping rate less than or equal to 5%.
6. The method for detecting elevator door position and open and close states according to claim 5 wherein the predetermined confidence threshold is 0.5 and the confidence formula for the elevator door frame is calculated as:
obj＝Clspred 0.3*Ctpred 0.7，
wherein, ClspredRepresenting each feature pointPredicted value of opening/closing state of elevator door, CtpredAnd (4) representing the predicted value of the distance between each characteristic point and the center position of the elevator door frame.
7. A detection device for elevator door position and opening and closing state is characterized by comprising a training module and a detection module,
the training module is used for training an elevator door detection network model based on a first-order full convolution target detection algorithm;
the detection module is used for acquiring a real-time image of the elevator door, and identifying the position of the elevator door and the opening and closing state of the elevator door in the real-time image of the elevator door based on the trained elevator door detection network model.
8. The elevator door position and switch state sensing device of claim 7 wherein the training module comprises:
the system comprises a preprocessing unit, a data processing unit and a data processing unit, wherein the preprocessing unit is used for randomly acquiring elevator door images of each elevator hall in multiple angles through a camera of a mobile robot, marking the positions of the elevator doors and the opening and closing states of the elevator doors in the elevator door images and establishing a training set;
the elevator door detection network model comprises a backbone network, a characteristic pyramid network and a full convolution detection head module which are sequentially connected, wherein the full convolution detection head module comprises a classification branch, a centrality branch and a square frame regression branch;
the initialization unit is used for initializing the network parameters of the elevator door detection network model to generate an initial weight and an initial bias;
the characteristic extraction unit is used for inputting all images of the training set into the initialized elevator door detection network model, extracting a characteristic diagram of the input image through the backbone network, and performing multi-scale characteristic fusion on the characteristic diagram through the characteristic pyramid network to generate characteristic diagrams with different scales;
the training unit is used for generating a size predicted value of each feature point on the feature map with different scales for regressing the elevator door frame through the frame regression branch, generating a nearness predicted value of each feature point from the center position of the elevator door frame through the center degree branch, generating an elevator door opening and closing state predicted value of each feature point through the classification branch, and calculating a loss value according to a preset loss function; and the weight and the bias of the elevator door detection network model are updated through repeated circulating forward propagation and backward propagation until a preset iteration stop condition is reached, and the trained elevator door detection network model is generated.
9. A computer-readable storage medium, storing a computer program, characterized in that the computer program, when being executed by a processor, carries out the steps of the method for detecting elevator door position and switch state according to any one of claims 1-6.
10. A terminal, characterized in that it comprises a computer-readable storage medium according to claim 9 and a processor which, when executing a computer program on the computer-readable storage medium, carries out the steps of the method for detecting elevator door position and switch state according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011570412.0A CN113435466B (en) | 2020-12-26 | 2020-12-26 | Method, device, medium and terminal for detecting elevator door position and opening and closing state |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011570412.0A CN113435466B (en) | 2020-12-26 | 2020-12-26 | Method, device, medium and terminal for detecting elevator door position and opening and closing state |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113435466A true CN113435466A (en) | 2021-09-24 |
| CN113435466B CN113435466B (en) | 2024-07-05 |
Family
ID=77752774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011570412.0A Active CN113435466B (en) | 2020-12-26 | 2020-12-26 | Method, device, medium and terminal for detecting elevator door position and opening and closing state |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113435466B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419451A (en) * | 2022-03-29 | 2022-04-29 | 北京云迹科技股份有限公司 | Method and device for identifying inside and outside of elevator, electronic equipment and storage medium |
| CN115063362A (en) * | 2022-06-10 | 2022-09-16 | 嘉洋智慧安全生产科技发展（北京）有限公司 | Distribution box door detection method, system, electronic device, medium, and program product |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
| CN110674866A (en) * | 2019-09-23 | 2020-01-10 | 兰州理工大学 | Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network |
| CN111046928A (en) * | 2019-11-27 | 2020-04-21 | 上海交通大学 | Single-stage real-time universal target detector with accurate positioning and method |
| CN111223088A (en) * | 2020-01-16 | 2020-06-02 | 东南大学 | Casting surface defect identification method based on deep convolutional neural network |
| CN111476252A (en) * | 2020-04-03 | 2020-07-31 | 南京邮电大学 | Computer vision application-oriented lightweight anchor-frame-free target detection method |
| CN111807183A (en) * | 2020-07-20 | 2020-10-23 | 北京电通慧梯物联网科技有限公司 | Elevator door state intelligent detection method based on deep learning |
| CN111814704A (en) * | 2020-07-14 | 2020-10-23 | 陕西师范大学 | Full convolution examination room target detection method based on cascade attention and point supervision mechanism |
| WO2020221990A1 (en) * | 2019-04-30 | 2020-11-05 | Huawei Technologies Co., Ltd. | Facial localisation in images |
-
2020
- 2020-12-26 CN CN202011570412.0A patent/CN113435466B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019192397A1 (en) * | 2018-04-04 | 2019-10-10 | 华中科技大学 | End-to-end recognition method for scene text in any shape |
| WO2020221990A1 (en) * | 2019-04-30 | 2020-11-05 | Huawei Technologies Co., Ltd. | Facial localisation in images |
| CN110674866A (en) * | 2019-09-23 | 2020-01-10 | 兰州理工大学 | Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network |
| CN111046928A (en) * | 2019-11-27 | 2020-04-21 | 上海交通大学 | Single-stage real-time universal target detector with accurate positioning and method |
| CN111223088A (en) * | 2020-01-16 | 2020-06-02 | 东南大学 | Casting surface defect identification method based on deep convolutional neural network |
| CN111476252A (en) * | 2020-04-03 | 2020-07-31 | 南京邮电大学 | Computer vision application-oriented lightweight anchor-frame-free target detection method |
| CN111814704A (en) * | 2020-07-14 | 2020-10-23 | 陕西师范大学 | Full convolution examination room target detection method based on cascade attention and point supervision mechanism |
| CN111807183A (en) * | 2020-07-20 | 2020-10-23 | 北京电通慧梯物联网科技有限公司 | Elevator door state intelligent detection method based on deep learning |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114419451A (en) * | 2022-03-29 | 2022-04-29 | 北京云迹科技股份有限公司 | Method and device for identifying inside and outside of elevator, electronic equipment and storage medium |
| CN114419451B (en) * | 2022-03-29 | 2022-06-03 | 北京云迹科技股份有限公司 | Method and device for identifying inside and outside of elevator, electronic equipment and storage medium |
| CN115063362A (en) * | 2022-06-10 | 2022-09-16 | 嘉洋智慧安全生产科技发展（北京）有限公司 | Distribution box door detection method, system, electronic device, medium, and program product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113435466B (en) | 2024-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110059558B (en) | Orchard obstacle real-time detection method based on improved SSD network | |
| CN106803071B (en) | Method and device for detecting object in image | |
| CN107833209B (en) | X-ray image detection method and device, electronic equipment and storage medium | |
| CN108960163B (en) | Gesture recognition method, device, equipment and storage medium | |
| CN109961107B (en) | Training method and device for target detection model, electronic equipment and storage medium | |
| CN111612002A (en) | Multi-target object motion tracking method based on neural network | |
| CN108629326A (en) | The action behavior recognition methods of objective body and device | |
| CN110246160B (en) | Video target detection method, device, equipment and medium | |
| CN113435466B (en) | Method, device, medium and terminal for detecting elevator door position and opening and closing state | |
| CN108182695B (en) | Target tracking model training method and device, electronic equipment and storage medium | |
| CN111353473B (en) | Face detection method and device, electronic equipment and storage medium | |
| CN110349138B (en) | Target object detection method and device based on example segmentation framework | |
| CN111738074B (en) | Pedestrian attribute identification method, system and device based on weak supervision learning | |
| CN110866428B (en) | Target tracking method, device, electronic equipment and storage medium | |
| CN111738403A (en) | Neural network optimization method and related equipment | |
| CN111950633A (en) | Neural network training method, neural network target detection method, neural network training device, neural network target detection device and storage medium | |
| CN111259838B (en) | Method and system for deeply understanding human body behaviors in service robot service environment | |
| CN113065379B (en) | Image detection method and device integrating image quality and electronic equipment | |
| CN117671548A (en) | Abnormal sorting detection method and device, electronic equipment and storage medium | |
| CN112906554B (en) | Model training optimization method and device based on visual image and related equipment | |
| CN111968102B (en) | Target equipment detection method, system, medium and electronic terminal | |
| CN112614168B (en) | Target face tracking method and device, electronic equipment and storage medium | |
| CN111382638A (en) | Image detection method, device, equipment and storage medium | |
| US20230334774A1 (en) | Site model updating method and system | |
| CN112308061B (en) | License plate character recognition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |