CN115973885A

CN115973885A - Object behavior determination method and device, electronic equipment and storage medium

Info

Publication number: CN115973885A
Application number: CN202211623903.6A
Authority: CN
Inventors: 唐其伟; 陈刚; 程伟; 杜卉然; 刘贤钊
Original assignee: Hitachi Building Technology Guangzhou Co Ltd
Current assignee: Hitachi Building Technology Guangzhou Co Ltd
Priority date: 2022-12-16
Filing date: 2022-12-16
Publication date: 2023-04-18

Abstract

The invention discloses a method and a device for determining object behaviors, electronic equipment and a storage medium, wherein the method for determining the object behaviors comprises the following steps: when a car of an elevator arrives at a waiting hall and a door is opened, a sensor is controlled to collect a plurality of frames of images in the area of the waiting hall; carrying out object detection tracking on the image to obtain at least two items of state data of each object in the elevator waiting hall; calculating the behavior score of the object according to the preset weight coefficient of each state data and the state data; determining that the behavior of the object is intended to be an elevator ride when the behavior score is greater than a preset score threshold. The embodiment solves the problem that misjudgment of the behavior intention is easy to occur when the behavior intention of the object is determined through the speed of the moving speed, the accuracy of determining the behavior intention of the object by synthesizing multiple items of state data of the object is high, the accuracy of identifying the object which takes the elevator intention outside the elevator door is improved, the accurate control of the door closing time of the elevator door is facilitated, and the operation efficiency of the elevator is improved.

Description

Object behavior determination method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of elevator control, in particular to a method and a device for determining object behaviors, electronic equipment and a storage medium.

Background

With the development of elevator intelligentization, a technology for sensing the movement trend of passengers in a waiting hall through a sensor and controlling the doors of an elevator to be closed or opened according to the movement trend of the passengers appears.

In the prior art, after multi-frame images of a hall are acquired by a sensor, the moving speed of people outside a door is analyzed by the multi-frame images, whether the people have the intention of taking an elevator or not is predicted by the moving speed, so that the door is controlled according to the intention of the people outside the door, however, the area outside the door is usually a public area, the people outside the door may be the elevator and also may pass through the door, the intention of the people is preset only according to the moving speed or the position, misjudgment is easily caused to the intention of the people, for example, people passing through the door without the elevator taking requirement are identified as people with the elevator taking intention, the prediction of the intention of the people is inaccurate, the control of the door is not facilitated, for example, people passing through the door without the elevator taking requirement are identified as people with the elevator taking intention, the door is delayed, and the operation efficiency of the door is reduced.

Disclosure of Invention

The invention provides an object behavior determination method, an object behavior determination device, electronic equipment and a storage medium, and aims to solve the problems that the intention of people outside an elevator door is predicted through the moving speed, so that the intention of the people is not predicted accurately and the elevator door is not controlled conveniently.

In a first aspect, the present invention provides an object behavior determining method, including:

when a car of an elevator arrives at a waiting hall and a door is opened, a sensor is controlled to collect a plurality of frames of images in the area of the waiting hall;

carrying out object detection tracking on the image to obtain at least two items of state data of each object in the elevator waiting hall;

calculating the behavior score of the object according to the preset weight coefficient of each state data and the state data;

determining that the behavior of the object is intended to be an elevator ride when the behavior score is greater than a preset score threshold.

In a second aspect, the present invention provides an object behavior determination apparatus, including:

the image acquisition module is used for controlling the sensor to acquire multi-frame images of the area of the elevator waiting hall when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened;

the object detection tracking module is used for carrying out object detection tracking on the image to obtain at least two items of state data of each object in the elevator waiting hall;

the behavior score acquisition module is used for calculating the behavior score of the object according to the preset weight coefficient of each state data and the state data;

and the behavior intention determining module is used for determining that the behavior intention of the object is to take an elevator when the behavior score is larger than a preset score threshold value.

In a third aspect, the present invention provides an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of determining object behavior of the first aspect of the invention.

In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a processor to implement the object behavior determination method according to the first aspect of the present invention when executed.

According to the embodiment of the invention, when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened, the control sensor collects multi-frame images in the area of the elevator waiting hall, the images are subjected to object detection and tracking to obtain at least two items of state data of each object in the elevator waiting hall, the behavior score of the object is calculated according to the preset weight coefficient and the state data of each item of state data, the behavior intention of the object is determined to be the elevator taking when the behavior score is larger than the preset score threshold value, the purpose that the behavior intention of the object is determined by obtaining at least two items of state data of each object through the images can be realized, for example, the behavior score of the object can be calculated after the position, the orientation, the moving speed and other data of the object are combined with the preset weight coefficient, the behavior intention of the object is determined to be the elevator taking when the behavior score is larger than the preset score threshold value, the problem that misjudgment of the behavior intention is easy to occur when the behavior intention of the object is determined through the speed of the moving speed is solved, the accuracy of determining the behavior intention of the object by integrating at least two items of state data of the object is high, the accuracy of the elevator door taking of the elevator is improved, and the door closing efficiency of the elevator door is improved.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic illustration of a sensor mounting and detection area;

fig. 2 is a flowchart of an object behavior determination method according to an embodiment of the present invention;

fig. 3A is a flowchart of an object behavior determination method according to a second embodiment of the present invention;

FIG. 3B is a schematic diagram of detection region division;

fig. 3C is a flow chart of controlling the doors during the process of the target object entering the car from the lobby;

FIG. 3D is a flow chart of adjusting a timer and controlling a door according to the timing of the timer;

fig. 4 is a schematic structural diagram of an object behavior determination apparatus according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

Example one

Fig. 2 is a flowchart of an object behavior determination method according to an embodiment of the present invention, where this embodiment is applicable to a situation where after a car arrives at a waiting hall and stops at an open door, a behavior intention of an object in the waiting hall is determined, and the method may be executed by an object behavior determination device, where the object behavior determination device may be implemented in a form of hardware and/or software, and the object behavior determination device may be configured in an electronic device, such as a controller of an elevator. As shown in fig. 2, the method for determining object behavior includes:

s201, when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened, the control sensor collects multi-frame images in the area of the elevator waiting hall.

As shown in fig. 1, in this embodiment, a sensor 3 is installed on the car of the elevator, and the sensor may be a general sensor, such as a black and white camera, an RGB camera, or an active light sensor, such as a TOF (Time of flight) depth sensor, a structured light sensor, or the like, that is, a sensor that obtains a depth image by sensing light emitted and received, of course, the sensor may also be a radar, and the present embodiment does not limit the type of the sensor.

In one example, the sensor of the present embodiment may be an active optical sensor, the sensor 3 may be mounted on the car 1 such that the detection area of the sensor 3 covers the area inside the car 1 and the area of the elevator hall 2, and optionally, the sensor 3 may be mounted on a lintel of a car door of the car 1, for example, at a middle position of the lintel of the car door, such that the detection area of the sensor 3 covers the area inside the car 1 and the area of the elevator hall 2.

In one example, the number of the sensors 3 may be one, that is, the sensors 3 may be active optical sensors having a large angle of view, further, when the number of the sensors 3 is one, the angle of the sensors 3 may be adjustable or fixed, or the angle of the light emitting axis of the sensors 3 with the vertical direction may be variable, that is, the sensors 3 may be adjustable in angle, so that the detection area 5 of the sensors 3 in the lobby 2 may be enlarged or reduced.

As shown in fig. 1, in another example, the number of the sensors 3 may be two, among the two sensors 3, the light emitting axis of one sensor 3 faces the car 1 to collect images of the area in the car 1, and the light emitting axis of the other sensor 3 faces the elevator waiting hall 2 to collect images of the area in the elevator waiting hall 2, optionally, the detection ranges of the two sensors 3 have an overlapping area to ensure that the images of all the areas from the car 1 to the elevator waiting hall 2 can be collected, and at the same time, when the sensors are 1 active light sensors in the vertical direction of the light emitting axes, the overexposure problem generated after the elevator is closed when the images of all the areas from the car 1 to the elevator waiting hall 2 are collected at the same time is avoided. For example, the angles of the light emission axes of the two sensors 3 with the vertical direction may be adjusted to adjust the detection areas of the two sensors 3. Of course, the angle of the light emission axis of the two sensors 3 to the vertical direction may also be fixed.

In practical application, when the elevator car of the elevator reaches the elevator waiting hall of the target floor in the ascending or descending process, the operation is stopped, and the elevator door is controlled to be opened, so that after passengers in the elevator car leave the elevator, objects with elevator taking requirements in the elevator waiting hall enter the elevator car, wherein the objects can be people, robots, pets and other objects needing to take the elevator.

When the elevator door is opened or after the elevator door is completely opened, the sensor 3 may acquire a depth image according to a preset frame rate and transmit the depth image to the controller of the elevator, as shown in fig. 1, when the number of the sensors 3 is one, the sensor 3 may be controlled to face the area of the elevator waiting hall 2 when the elevator door starts to be opened so as to acquire multi-frame images, and when the number of the sensors 3 is two, the sensor facing the elevator waiting hall 2 may be controlled to acquire multi-frame images of the area of the elevator waiting hall 2, it should be noted that the acquisition of the images by the sensor 3 may start from the opening of the elevator door and end when the elevator door is closed.

S202, carrying out object detection tracking on the image to obtain at least two items of state data of each object in the elevator waiting hall.

In this embodiment, the object may be a person, a pet, a robot, or the like, and the object detection tracking model may be trained in advance, and after the image sequence is input, the object detection tracking model may identify an object in each frame of image in the image sequence and predict state data such as a position, a moving speed, a moving direction, a contour volume size, a moving track, or the like of each object, where the training method of the object detection tracking model may refer to an existing training method of the object detection tracking model, and will not be described in detail herein.

After the images are acquired, the image sequence may be generated according to the acquisition time of the images, and the image sequence is input into a pre-trained object detection and tracking model, so that each object located in the elevator waiting hall and state data such as the position, the moving speed, the contour volume, the orientation and the like of each object in the elevator waiting hall are identified through the object detection and tracking model.

The position may be a distance from the landing door to the object in the landing hall, and the moving speed may be a moving speed of the object in the landing hall, for example, taking the object as an example, when the landing door is opened, the person moves from the landing hall to the landing door, and the contour volume may be an outer contour volume of the object identified by each image, for example, when the person is stationary and does not intend to take the elevator, the feet of the person do not move, the hands of the person do not swing, the outer contour volume is small, when the person moves to the landing door, the feet of the person move, the hands of the person swing, the outer contour volume is large, the orientation may be an orientation of the identified object, the orientation generally indicates a moving direction of the object, the object is taken as an example, the orientation may be a face direction of the person, in this embodiment, the orientation is a positive angle toward the landing door, and an included angle with a plane where the landing door is located is a size of the orientation.

And S203, calculating the behavior score of the object according to the preset weight coefficient and the state data of each state data.

In this embodiment, a weighting coefficient table may be preset for each item of status data, and in the weighting coefficient table, a corresponding weighting coefficient may be searched according to a numerical value of each item of status data, and of course, the weighting coefficient of each item of status data may also be calculated according to a predetermined rule.

After the weight coefficient of each item of state data is acquired, a weighted sum can be calculated by using a plurality of items of state data and corresponding weight coefficients, that is, a behavior score of the object can be obtained, the behavior score represents the possibility that the object takes the elevator, and the higher the behavior score is, the higher the possibility that the object takes the elevator is.

And S204, when the behavior score is larger than a preset score threshold value, determining that the behavior intention of the object is to take an elevator.

The embodiment can set a score threshold, when the behavior score of the object is greater than the preset score threshold, the object is determined to be in the elevator with a high probability, the behavior intention of the object can be determined to be in the elevator, so that the elevator door can be controlled according to whether the behavior intention of the object in the elevator waiting hall is in the elevator, exemplarily, after the behavior intentions of all objects in the elevator waiting hall are determined, the object whose behavior is intended to be in the elevator is tracked through the image acquired by the sensor, after the object whose behavior is intended to be in the elevator enters the elevator car, if the closing time of the elevator door is not reached, the elevator door can be controlled to be closed in advance, the elevator door does not need to be closed manually in the elevator car, or the elevator door does not need to be closed when the object whose behavior is intended to be in the elevator car is tracked to be in the closing time, so that the elevator door is closed after the object with the elevator enters the elevator car, and the situation that the object whose behavior is closed when the elevator door does not enter the elevator car in time, the elevator car is avoided that the object whose behavior intention is closed, and even the object is in the process of closing the elevator door.

According to the embodiment of the invention, when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened, the control sensor collects multi-frame images in the area of the elevator waiting hall, the images are subjected to object detection and tracking to obtain at least two items of state data of each object in the elevator waiting hall, the behavior score of the object is calculated according to the preset weight coefficient and the state data of each item of state data, the behavior intention of the object is determined to be an elevator when the behavior score is larger than the preset score threshold value, the purpose that the behavior intention of the object is taken when the behavior score is larger than the preset score threshold value is achieved by obtaining at least two items of state data of each object through the images to determine the behavior of the object, for example, the behavior score of the object can be calculated after the position, the orientation, the moving speed and other data of the object are combined with the preset weight coefficient, the behavior intention of the object is determined to be the elevator when the behavior score is larger than the preset score threshold value, the problem that misjudgment of the behavior intention of the behavior of the object is easy to occur is determined through the speed of the moving speed is solved, the object is determined by integrating the plurality of the state data of the object, the accuracy of the behavior intention of determining the object, the accuracy of the object of taking the object is improved, the accuracy of the taking of the elevator door is facilitated, and the efficiency of controlling the elevator door closing of the elevator is improved.

Example two

Fig. 3A is a flowchart of an object behavior determination method provided in the second embodiment of the present invention, and the embodiment of the present invention performs optimization based on the first embodiment, as shown in fig. 3A, the object behavior determination method includes:

s301, when the elevator car of the elevator arrives at the elevator waiting hall, the door opening distance of the elevator door of the elevator is detected.

When the elevator car of the elevator goes up or goes down, the elevator car stops running when reaching a waiting hall of a target floor, and the elevator door is controlled to be opened so that passengers in the elevator car leave the elevator, and an object which is required by the elevator in the waiting hall enters the elevator car, the door opening distance of the elevator door can be detected, wherein the door opening distance refers to the opening degree of the door when the elevator door is opened.

In one example, the door opening distance of the elevator door can be acquired from the door controller, and after the door edge is identified through the image acquired by the sensor, the door opening distance can be calculated through the combination of the pixel where the door edge is located and the installation parameters, imaging principles and the like of the sensor.

Of course, the door opening distance of the ladder door can be detected by the distance sensor installed on the ladder door, and the method for detecting the door opening distance is not limited in the embodiment.

S302, when the door opening distance is larger than a preset distance threshold value, controlling a sensor with the shooting direction facing a lift waiting hall to collect multi-frame images according to a preset frame rate to obtain an image sequence.

The preset distance threshold value can be determined according to the field angle of the sensor, the width of the ladder door and the installation position of the sensor, and the preset distance threshold value is not limited in the embodiment, so that the proportion of the elevator waiting hall in the image acquired by the sensor through the gap opened by the ladder door is larger than the preset proportion.

When the door opening distance is larger than a preset distance threshold value, if the number of the sensors is one, the sensors are controlled to face the elevator waiting hall to acquire images, and if the number of the sensors is two, the sensors facing the elevator waiting hall are controlled to acquire images, wherein the frame rate of the images acquired by the sensors can be a fixed frame rate or a dynamically adjustable frame rate.

After the sensors acquire images, an image sequence may be generated according to the acquisition sequence of the images, it should be noted that when the number of the sensors is two, the images acquired by the two sensors may be spliced to obtain a spliced image, and the spliced image is used to generate the image sequence.

This embodiment control sensor when the distance of opening the door of ladder is greater than preset distance threshold value according to preset frame rate collection image formation image sequence, avoids the sensor to continuously gather the image and gather invalid image when the distance of opening the door is less, on the one hand, can reduce sensor collection image quantity, prolongs the life of sensor, and on the other hand can reduce image data's handling capacity.

S303, inputting the image sequence into a pre-trained object detection tracking model to obtain at least two items of state data of each object, wherein each item of state data of each object comprises the orientation, the position, the volume and the moving speed of the object.

In this embodiment, an image sequence may be generated after binarization is performed on an image acquired by a sensor, and the image sequence is input into the object detection and tracking model to obtain a direction, a position, a volume, and a moving speed of each object, which may also include other state data such as a moving track of each object.

As shown in fig. 1, taking the human being as an example, the orientation data is an angle θ between the face direction of the human being and the plane where the door 4 is located, the position is a vertical distance of the human being from the door 4 in the lobby, the volume may be a volume of the detected outline of the human being, and the volume when the human is at rest is smaller than the volume when the human is walking.

In the present embodiment, the object detection tracking model may be a neural network such as RNN, CNN, DNN, etc., and when the object detection tracking model is trained, the image with the direction, position, volume, and moving speed of the object in the image labeled is used as training data, the image sequence after labeling is input to the object detection tracking model, the direction, position, volume, and moving speed of each object in the image sequence are predicted, the loss rate is calculated from the predicted data and the labeled data, and the model parameter is adjusted by the loss rate until the loss rate is smaller than a preset value, so as to obtain the trained object detection tracking model.

And S304, searching a direction weight coefficient matched with the orientation of the object in a preset orientation-direction weight coefficient table.

As shown in fig. 1, different orientations of objects can represent the intention of the object to ride on an elevator, in fig. 1, objects a and D are oriented toward the landing door 4, and object B is oriented parallel to the landing door, it can be determined that objects a and D are more likely to ride on an elevator relative to object B, object C is oriented at an angle θ less than 90 ° to the landing door 4, and object C is more likely to ride on an elevator than object B and less than objects a and D.

Specifically, in the embodiment of the present invention, weights corresponding to different orientations may be preset to generate an orientation-direction weight coefficient table, which is exemplarily shown in table 1 below:

table 1:

orientation range theta	Coefficient of directional weight
		θ∈(0，30°]	0.2
θ∈(30°，65°]	0.7
		θ∈(65°，90°]	0.9

Table 1 is merely an example, and those skilled in the art may also set different orientation ranges and corresponding direction weight coefficients, which is not limited in this embodiment of the present invention.

In another alternative embodiment, the corresponding directional weight coefficients may also be calculated according to the orientation, and exemplarily, the directional weight coefficients are positively correlated with the orientation, so that the corresponding directional weight coefficients may be dynamically calculated according to different orientations.

S305, determining a position weight coefficient of the object according to the position of the object.

As shown in fig. 1, objects are different in distance from the doors and different in probability of taking an elevator, and generally, objects closer to the doors 4 are more likely to take an elevator, and in fig. 1, objects a and C are closer to the doors 4 and object B is farther from the doors 4, it can be determined that objects a and C are more likely to take an elevator relative to object B, and therefore, a position-position weight coefficient table can be set in advance by looking up a position weight coefficient corresponding to the position of an object in the position-position weight coefficient table.

In another alternative embodiment, the position weight coefficient may also be calculated by the position, and the position weight coefficient is larger the closer the position is to the door, i.e. the position weight coefficient is positively correlated with the position.

And S306, calculating the swing amplitude of the object based on the volume.

The volume referred to in this embodiment is the volume of the detected outer contour of the object, when the object is stationary, both feet and both hands of the object are almost stationary, taking the object as a person as an example, when a door is opened, a passenger with an elevator riding intention moves into a car from a lobby, both hands of the passenger swing, both feet alternately step forward, the outer contour of the passenger changes, and the outer contour volume of the passenger also changes, so that the maximum volume and the minimum volume can be found out from the volumes of the passengers determined by multiple frames of images, the swing amplitude is calculated by the maximum volume and the minimum volume, exemplarily, when a door opening sensor acquires a first frame of image, the passenger riding the elevator needs to wait for the passenger in the car to move into the elevator after leaving the elevator, at this time, the passenger is stationary, the outer contour volume of the passenger can be minimum, when the passenger moves into the car, both feet of the outer contour volume becomes large, the difference between the maximum volume and the minimum volume can be calculated, and the ratio of the difference to the minimum volume is calculated as the swing amplitude.

In another optional embodiment, the swing amplitude of the object may also be measured according to the number of pixels occupied by the object in each frame of image in the image sequence, and the embodiment does not limit the way of calculating the swing amplitude of the object.

And S307, searching a swing amplitude weight coefficient matched with the swing amplitude in a preset swing amplitude-swing amplitude weight coefficient table.

The larger the swing amplitude of the object is, the larger the probability that the object in the elevator waiting hall moves after the elevator door is opened is, the larger the swing amplitude weight coefficient corresponding to the swing amplitude is, and the amplitude weight coefficient matched with the swing amplitude can be searched in a preset swing amplitude-swing amplitude weight coefficient table.

And S308, searching a speed weight coefficient matched with the moving speed of the object in a preset moving speed-speed weight coefficient table.

When an object takes an elevator when a door is opened, passengers usually enter the elevator quickly because the door opening time is limited, namely the object takes the elevator more obviously the faster the object goes to the door, and the larger the speed weight coefficient corresponding to the speed of the object is, the speed weight coefficient matched with the moving speed of the object can be searched from the moving speed-speed weight coefficient table.

S309, calculating a weighted sum by using the orientation, the direction weight coefficient, the position weight coefficient, the swing amplitude weight coefficient, the moving speed and the speed weight coefficient to serve as the behavior score of the object.

In an alternative embodiment, the behavior score may be calculated by the following formula:

S＝r ₁ ×a+r ₂ ×b+r ₃ ×c+r ₄ ×d

in the above formula, r ₁ 、r ₂ 、r ₃ 、r ₄ The direction weight coefficient, the position weight coefficient, the swing amplitude weight coefficient and the speed weight coefficient are respectively used, and the a, b, c and d are respectively the direction, the position, the swing amplitude and the moving speed.

In another embodiment, the sum of the direction weight coefficient, the position weight coefficient, the swing amplitude weight coefficient, and the velocity weight coefficient may also be directly calculated as the behavior score of the object.

And S310, when the behavior score is larger than a preset score threshold value, determining that the behavior of the object intends to take the elevator.

The embodiment may set a score threshold, determine that the object will take the elevator with a high probability when the behavior score of the object is greater than the preset score threshold, determine that the behavior of the object is intended to take the elevator, perform tracking detection on the object whose behavior is intended to take the elevator, and determine that the object does not have a demand for taking the elevator when the behavior score is less than the preset score threshold.

And S311, judging whether the target object with the behavior intention of taking the elevator exists in the elevator waiting hall or not according to the current image.

Optionally, target detection and tracking may be performed on target objects whose behaviors intend to take the elevator, a position of each target object in the current image is obtained, and when the positions of the target objects are all located in the car, it is determined that all the objects whose behaviors intend to take the elevator in the elevator waiting hall have entered the car, S313 is performed, otherwise, S312 is performed.

And S312, controlling the elevator door according to the position of the target object.

In an optional embodiment, whether a target object located in a preset door-to-door area is detected in a current image or not can be judged, if yes, the elevator door is controlled to be kept open, countdown of a countdown timer is started, a next frame image is obtained to serve as the current image, whether the target object located in the door-to-door area of the elevator door is detected in the current image or not is judged continuously, if not, whether the target object located in a preset door-to-door area on the car side is detected in the current image or not is judged, when the target object located in the door-to-door area is detected, the distance from the target object to the door-to-door area is obtained, timing of the countdown timer is adjusted according to the distance, the elevator door is controlled according to timing, when the target object located in the door-to-door area is not detected, the next frame image is obtained to serve as the current image, and the step of judging whether the target object which is used for taking the elevator in the elevator hall according to the current image or not is carried out.

When the distance is less than or equal to the preset distance threshold, the first value (for example, 1 second time is subtracted) from the timing of the countdown timer, and when the distance is greater than the preset distance threshold, the second value (for example, 2 seconds is subtracted) from the timing of the countdown timer.

When the elevator door is controlled according to the timing of the timer, whether the timing of the countdown timer is greater than 0 or not can be judged, if yes, the elevator door is kept open, the next frame of image is obtained to serve as the current image, the step of judging whether the target object located in the preset door approaching area on the elevator car side is detected in the current image is returned, and if not, the step of judging whether the target object with the behavior intention of taking the elevator exists in the elevator waiting hall or not is returned according to the current image.

In order to make the person skilled in the art understand more clearly the control process of the elevator door in the process of the object in the elevator waiting hall entering the car, the following describes the process of controlling the closing of the elevator door with reference to the accompanying drawings and examples, and for convenience of description, as shown in fig. 3B, the detection area is divided into an inter-door area P1, a near-door area P2 and an elevator waiting hall area P3, where, the inter-door area P1 refers to an area where the elevator door is opened or closed, for example, a width area formed by 10cm on each side of the track when the elevator door is opened or closed, the near-door area P2 may be a width area formed by 15cm on the inner side and 30cm on the outer side with the center of the track of the elevator door, the elevator waiting hall area P3 is an area of the elevator waiting hall, and as shown in fig. 3C, the door control process in the process of the object a with the intention of taking the elevator entering the position a from the position A0 of the elevator waiting hall to the position A3 in the car is as follows:

s0, the ladder door is completely opened.

S1, detecting whether a target object exists in the door area P1 through a current image, if so, executing S2, and if not, executing S3.

The present embodiment may establish a target object sequence, where the target object sequence includes various items of data of each target object, such as at least a position of the target object.

As shown in fig. 3B, if the target object a enters the position A1 of the interdoor area P1 from the position A0 of the lobby, it is determined that the target object a exists in the interdoor area P1, S2 is performed, otherwise S3 is performed.

S2, when the target object exists in the door-to-door area P1, the control ladder door is kept open, the timer starts counting down, the next frame of image is obtained, and the process returns to S1.

In this embodiment, if there is a target object in the inter-door area P1 intended to be taken by the elevator, the initialization timer does not count down the elevator, and because there is a target object in the inter-door area, it is necessary to keep the door open, that is, a close signal is not generated, and the next frame of image is continuously acquired and returned to S1, and it is continuously determined whether there is a target object in the inter-door area.

And S3, detecting whether a target object exists in the approach area P2 through the current image, if so, executing S4, and if not, executing S5.

Specifically, as shown in fig. 3B, if the target object a is not detected in the inter-door area P1, it indicates that the target object a is still in the hall area P3, the next frame of image is continuously acquired, and the image is returned to S1, or the target object a has already entered the position A3 in the car 1, the tracking of the target object a is ended, if the target object is detected in the near-door area P2 on the car 1 side, it indicates that the target object has entered the near-door area on the car 1 side from the inter-door area P1, for example, the target object a moves from the position A1 to the position A2, S4 is executed, otherwise S7 is executed.

S4, a target object exists in the near door area P2, and the distance from the target object in the near door area P2 to the door area P1 is obtained;

and S5, adjusting the timing of the countdown timer according to the distance.

Specifically, as shown in fig. 3D, S5 may include the following sub-steps:

s51, judging whether the distance is smaller than a preset distance threshold value or not;

illustratively, the distance threshold is 10cm, if the distance is less than or equal to 10cm, S52 is executed, otherwise, S53 is executed.

S52, subtracting the first value from the counting down time of the timer.

Illustratively, when each frame of image detects that the distance from the target object in the near door area P2 to the door area P1 is less than or equal to 10cm, the count-down timer counts down for 1 second, i.e. the target object just leaves the door area, namely the door is closed immediately, but there is a risk of closing the door immediately, and meanwhile, in order to avoid the waiting time when the timer needs to count down to 0 being too long, the count-down timer may be reduced by 1 second.

And S53, subtracting a second value from the counting down time of the timer.

Illustratively, when each frame of image detects that the target object located in the near portal area P2 is more than 10cm to the interportal area P1, the count-down timer counts down by 2 seconds. That is, when the target object passes through the door zone and enters the near door zone on the car side at a longer distance, the risk of the door closing clamp to the target object is reduced, and when the target object is detected in the near door zone in one frame of image and the distance from the door zone is greater than 10cm, the door is closed immediately, but there is a risk of the door closing, and in order to avoid an excessively long waiting time when the waiting timer needs to count down to 0, the timer may be reduced by 2 seconds to shorten the waiting time for closing the door, and of course, when the target object is detected in the door zone and other target objects are simultaneously detected, the countdown timer is reset and starts counting down again.

And S6, controlling the elevator door according to the timing.

Specifically, as shown in fig. 3D, S6 may include the following sub-steps:

s61, judging whether the timing counter is greater than 0, if so, executing S62, and if not, executing S63;

s62, keeping the door open, acquiring the next frame of image as the current image, and returning to S1;

and S63, when the current image detects that no target object exists in the elevator waiting hall area, controlling the elevator door to be closed.

And S7, when the target object in the near door area is not detected, acquiring the next frame of image as the current image, and returning to S1.

It should be noted that, when receiving the current image collected by the sensor, first executing step S311, and if determining to execute step S312 according to step S311, executing step S1-S7, that is, in this embodiment, when receiving one frame of image, first determining whether there is any target object in the elevator waiting hall for taking an elevator, if there is no target object for taking an elevator, closing the elevator door, if there is any target object for taking an elevator, executing step S1-S7, resetting the timer countdown when it is detected that the target object is in the inter-door area in each frame of image, and adjusting the timing of the timer according to the calculated distance of the target object in the near-door area on the car side, when the timing is greater than 0, not closing the door, acquiring the next frame of image, and when the timer is less than or equal to 0, if there is no target object intended for taking an elevator in the elevator waiting hall, controlling the elevator door to close, so that the closing time of the elevator door can be dynamically adjusted according to the position of the target object for taking an elevator in the process of entering the car.

And S313, controlling the ladder door to be closed.

Because the car stops when waiting for the ladder room at the target floor and opens the door, the passenger in the car walks out the car earlier, waits for in the ladder room to have the passenger who takes advantage of the ladder demand just to get into the car, if confirm that the object that has the demand of taking advantage of the ladder has all got into the car in waiting for the ladder room, then can control the door and close, need not to wait for to arrive the time of closing the door of predetermineeing, improves the operating efficiency of elevator.

In the embodiment, an image sequence acquired by a sensor is input into a pre-trained object detection tracking model to obtain the orientation, position, volume and moving speed of each object, a weighted sum is calculated after the weight coefficients of the orientation, position, volume and moving speed are obtained and is used as the behavior score of the object, when the behavior score is greater than a preset score threshold value, the behavior intention of the object is determined to be an elevator, the problem that misjudgment of the behavior intention is easy to occur when the behavior intention of the object is determined according to the speed of the moving speed is solved, the accuracy of determining the behavior intention of the object by integrating the orientation, position, volume and moving speed of the object is high, the accuracy of identifying the object which is outside the elevator door and is used for taking the elevator is improved, the door closing time of the elevator door is controlled accurately, and the operation efficiency of the elevator is improved.

Furthermore, the situation that no target object with the behavior intention of taking the elevator exists in the elevator waiting hall is judged according to the current image, the elevator door is controlled to be closed, the preset door closing time does not need to be waited for, and the operation efficiency of the elevator is improved.

Furthermore, when the target object with the behavior intention of taking the elevator is judged to exist in the elevator waiting hall according to the current image, the countdown of the timer is started when the target object is in the door area, and the timing of the timer is shortened when the target object enters the near door area on the car side from the door area, namely, the timing of the timer is dynamically adjusted according to each position of the target object entering the car from the elevator waiting hall, and finally, the elevator door is controlled to be closed when the timer is equal to 0 and the target object without the elevator taking intention exists in the elevator waiting hall, so that the elevator door can be flexibly controlled to be closed.

EXAMPLE III

Fig. 4 is a schematic structural diagram of an object behavior determination apparatus according to a third embodiment of the present invention. As shown in fig. 4, the object behavior determination apparatus includes:

the image acquisition module 401 is used for controlling the sensor to acquire a plurality of frames of images of the area of the elevator waiting hall when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened;

an object detection and tracking module 402, configured to perform object detection and tracking on the image to obtain at least two pieces of state data of each object in the elevator waiting hall;

a behavior score obtaining module 403, configured to calculate a behavior score of the object according to a preset weight coefficient of each state data and the state data;

a behavior intention determining module 404, configured to determine that the behavior intention of the object is to take an elevator when the behavior score is greater than a preset score threshold.

Optionally, the image acquisition module 401 includes:

the door opening distance detection unit is used for detecting the door opening distance of a door of the elevator when the elevator car of the elevator arrives at the elevator waiting hall;

and the image acquisition unit is used for controlling a sensor which shoots towards the elevator waiting hall to acquire multi-frame images according to a preset frame rate to obtain an image sequence when the door opening distance is greater than a preset distance threshold value.

Optionally, the object detection and tracking module 402 includes:

the image sequence generating unit is used for generating an image sequence by the acquired multi-frame images according to the acquisition time of the images;

and the model input unit is used for inputting the image sequence into a pre-trained object detection tracking model to obtain at least two items of state data of each object.

Optionally, the status data comprises at least two of orientation, position, volume, speed of movement of each object.

Optionally, the behavior score obtaining module 403 includes:

a direction weight coefficient acquisition unit for looking up a direction weight coefficient matching the orientation of the object in a preset orientation-direction weight coefficient table;

a position weight coefficient obtaining unit, configured to determine a position weight coefficient of the object according to the position of the object;

a swing amplitude calculation unit for calculating a swing amplitude of the object based on the volume;

the swing amplitude weight coefficient acquisition unit is used for searching a swing amplitude weight coefficient matched with the swing amplitude in a preset swing amplitude-swing amplitude weight coefficient table;

a speed weight coefficient acquisition unit for searching a speed weight coefficient matching the moving speed of the object in a preset moving speed-speed weight coefficient table;

and the behavior score calculating unit is used for calculating a weighted sum by adopting the orientation, the direction weight coefficient, the position weight coefficient, the swing amplitude weight coefficient, the moving speed and the speed weight coefficient to serve as the behavior score of the object.

Optionally, the method further comprises:

the target object judging module is used for judging whether a target object with an action intention of taking an elevator exists in the elevator waiting hall according to the current image;

the first elevator door control module is used for controlling the elevator door according to the position of the target object;

and the second elevator door control module is used for controlling the elevator door to be closed.

Optionally, the target object determining module includes:

the position tracking unit is used for carrying out target detection tracking on the target object to obtain the position of the target object in the current image;

the first determining unit is used for determining that the target object which is used for taking an elevator and does not exist in the elevator waiting hall when the positions of the target objects are all located in the elevator car;

and the second determining unit is used for determining that the target object with the behavior intention of taking an elevator exists in the elevator waiting hall when the position of the target object is positioned outside the elevator door.

Optionally, the first door control module comprises:

the inter-door area judging subunit is used for judging whether a target object located in a preset inter-door area is detected in the current image;

the door opening keeping control subunit is used for controlling the door to be opened, starting a countdown timer to count down, acquiring the next frame of image as the current image, and returning to the inter-door area judgment subunit;

the near door area judging subunit is used for judging whether a target object located in a near door area preset on the car side is detected in the current image;

the distance acquisition subunit is used for acquiring the distance from the target object to the interdoor area when the target object positioned in the near door area is detected;

the timer adjusting subunit is used for adjusting the timing of the countdown timer according to the distance and controlling the elevator door according to the timing;

and the image acquisition subunit is used for acquiring the next frame of image as the current image and returning the current image to the target object judgment module when the target object in the near door area is not detected.

A terminal, the timer adjustment subunit including:

the distance judging component is used for judging whether the distance is smaller than a preset distance threshold value or not;

a first timer adjustment component for subtracting a first value from the count-down timer when the distance is less than or equal to the preset distance threshold;

a second timer adjustment component configured to subtract a second value from the count down timer if the distance is greater than the predetermined distance threshold.

Optionally, the timer adjustment subunit includes:

the timing judgment component is used for judging whether the timing of the countdown timer is greater than 0;

the ladder door keeping and opening control assembly is used for controlling the ladder door to keep open, acquiring the next frame of image as the current image and returning to the inter-door area judgment subunit;

and the image acquisition component is used for acquiring the next frame of image and returning the next frame of image to the target object judgment module.

The object behavior determination device provided by the embodiment of the invention can execute the object behavior determination method provided by the first embodiment and the second embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Example four

FIG. 5 illustrates a schematic diagram of an electronic device 50 that may be used to implement an embodiment of the invention. The electronic device 50 is intended to represent various forms of digital computers, such as desktop computers, workstations, servers, blade servers, mainframe computers, and the like. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 5, the electronic device 50 includes at least one processor 51, and a memory communicatively connected to the at least one processor 51, such as a Read Only Memory (ROM) 52, a Random Access Memory (RAM) 53, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 51 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 52 or the computer program loaded from a storage unit 58 into the Random Access Memory (RAM) 53. In the RAM 53, various programs and data necessary for the operation of the electronic apparatus 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input/output (I/O) interface 55 is also connected to bus 54.

A plurality of components in the electronic apparatus 50 are connected to the I/O interface 55, including: an input unit 56 such as a keyboard, a mouse, a sensor, and the like; an output unit 57 such as various types of displays, speakers, and the like; a storage unit 58 such as a magnetic disk, optical disk, or the like; and a communication unit 59 such as a network card, modem, wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 51 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processors 51 include, but are not limited to, central Processing Units (CPUs), graphics Processing Units (GPUs), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processors, controllers, microcontrollers, and the like. The processor 51 performs the various methods and processes described above, such as the object behavior determination method.

In some embodiments, the object behavior determination method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 50 via the ROM 52 and/or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the object behavior determination method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the object behavior determination method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired result of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An object behavior determination method is applied to determining the behavior of an object in a lobby, and comprises the following steps:

2. The method of claim 1, wherein the controlling the sensor to collect a plurality of frames of images of the area of the waiting hall when the car of the elevator arrives at the waiting hall and the doors are opened comprises:

when a lift car of the elevator arrives at a waiting hall, detecting the door opening distance of a lift door of the elevator;

and when the door opening distance is greater than a preset distance threshold value, controlling a sensor with a shooting direction facing the elevator waiting hall to collect multi-frame images according to a preset frame rate to obtain an image sequence.

3. The method of claim 1, wherein said performing object detection tracking on said image to obtain at least two items of status data for each object in said lobby comprises:

generating an image sequence by the acquired multi-frame images according to the acquisition time of the images;

and inputting the image sequence into a pre-trained object detection tracking model to obtain at least two items of state data of each object.

4. The method of claim 1, wherein the state data includes at least two of orientation, position, volume, speed of movement of each object.

5. The method of claim 1, wherein the state data includes orientation, position, volume, and speed of movement of the object, and the calculating the behavioral score of the object from the preset weight coefficient for each state data and the state data comprises:

looking up a direction weight coefficient matched with the orientation of the object in a preset orientation-direction weight coefficient table;

determining a position weight coefficient of the object according to the position of the object;

calculating a swing amplitude of the object based on the volume;

searching a swing amplitude weight coefficient matched with the swing amplitude in a preset swing amplitude-swing amplitude weight coefficient table;

searching a speed weight coefficient matched with the moving speed of the object in a preset moving speed-speed weight coefficient table;

and calculating a weighted sum by using the orientation, the direction weight coefficient, the position weight coefficient, the swing amplitude weight coefficient, the moving speed and the speed weight coefficient to be used as the behavior score of the object.

6. The method of any one of claims 1-5, further comprising, after determining that the behavior of the object is intended for riding an elevator:

judging whether a target object with behavior intention of taking an elevator exists in the elevator waiting hall according to the current image;

if so, controlling the ladder door according to the position of the target object;

if not, controlling the ladder door to close.

7. The method of claim 6, wherein said determining from the current image whether there are any more target objects in the lobby that are intended for elevator riding, comprises:

carrying out target detection tracking on the target object to obtain the position of the target object in the current image;

when the positions of the target objects are all located in the elevator car, determining that no target object which is used for taking an elevator and is supposed to be in the elevator waiting hall exists;

and when the position of the target object is positioned outside the elevator door, determining that the target object with the behavior intention of taking an elevator exists in the elevator waiting hall.

8. The method of claim 6, wherein said controlling the landing doors based on the position of the target object comprises:

judging whether a target object located in a preset interdoor area is detected in the current image;

if yes, controlling the ladder door to be kept open, starting a countdown timer to count down, acquiring a next frame of image as a current image, and returning to the step of judging whether a target object located in an inter-door area of the ladder door is detected in the current image;

if not, judging whether a target object located in a preset door approaching area on the car side is detected in the current image;

when a target object located in the near door area is detected, the distance from the target object to the area between the doors is obtained;

adjusting the timing of the countdown timer according to the distance, and controlling the elevator door according to the timing;

and when the target object in the near door area is not detected, acquiring the next frame of image as a current image, and returning to the step of judging whether the target object with the behavior intention of taking the elevator exists in the elevator waiting hall according to the current image.

9. The method of claim 8, wherein said adjusting the timing of said countdown timer based on said distance comprises:

judging whether the distance is smaller than a preset distance threshold value or not;

subtracting a first value from the count down timer when the distance is less than or equal to the preset distance threshold;

and when the distance is larger than the preset distance threshold value, subtracting a second numerical value from the timing of the countdown timer.

10. The method of claim 8, the controlling the door according to the timing, comprising:

judging whether the timing of the countdown timer is greater than 0;

if yes, controlling the elevator door to be kept open, acquiring a next frame of image as a current image, and returning to the step of judging whether a target object located in a door approaching area preset on the car side is detected in the current image;

and if not, acquiring the next frame of image, and returning to the step of judging whether the behavior intention is the target object taking the elevator in the elevator waiting hall according to the current image.

11. An object behavior determination device, applied to determining the behavior of an object in a lobby, comprises:

the image acquisition module is used for controlling the sensor to acquire multi-frame images of the region of the elevator waiting hall when the elevator car of the elevator arrives at the elevator waiting hall and the elevator door is opened;

and the behavior intention determining module is used for determining that the behavior intention of the object is to take an elevator when the behavior score is larger than a preset score threshold.

12. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the object behavior determination method of any one of claims 1-10.

13. A computer-readable storage medium storing computer instructions for causing a processor to perform the method of determining object behavior of any one of claims 1-10 when executed.