WO2022134504A9

WO2022134504A9 - Image detection method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022134504A9
Application number: PCT/CN2021/101619
Authority: WO
Inventors: 林少波; 曾星宇; 赵瑞
Original assignee: 上海商汤智能科技有限公司
Priority date: 2020-12-25
Filing date: 2021-06-22
Publication date: 2022-07-21
Also published as: CN114283305A; CN112560986A; KR20220095218A; JP2023510477A; CN112560986B; WO2022134504A1

Abstract

An image detection method and apparatus, an electronic device, and a storage medium. The method comprises: obtaining a first image of an escalator (S11); performing region detection on the first image to determine that the escalator is in a first region of the first image (S12); performing escalator state recognition on a first region image that corresponds to the first region to determine at least one state recognition result of the escalator (S13), the state recognition result comprising that the escalator is in an empty state or in a non-empty state; and determining the state of the escalator according to the at least one state recognition result (S14).

Description

Image detection method and device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is based on a Chinese patent application with application number 202011559951.4 and an application date of December 25, 2020, and claims the priority of the Chinese patent application, the entire contents of which are hereby incorporated into the present disclosure by full text .

technical field

The present disclosure relates to the technical field of image detection and the technical field of computer vision, and in particular, to an image detection method and apparatus, an electronic device and a storage medium.

Background technique

With the development of the economy and the continuous improvement of infrastructure construction, the application of escalators in shopping malls, office buildings, public transportation and other scenarios is becoming more and more extensive. During the daily operation of the escalator, people need to strictly manage the area where the escalator is located to ensure the operation of the elevator and avoid accidents. The automatic detection scheme for the elevator area in the related art is usually based on infrared induction or gravity induction to realize the start and stop of the elevator, which is prone to misidentification and poor detection effect.

SUMMARY OF THE INVENTION

The embodiment of the present disclosure proposes an image detection technical solution.

According to an aspect of the embodiments of the present disclosure, there is provided an image detection method, including: acquiring a first image of an escalator; performing area detection on the first image, and determining an image of the escalator in the first image. The first area; perform elevator state recognition on the image of the first area corresponding to the first area, and determine at least one state recognition result of the escalator, where the state recognition result includes that the escalator is in an empty state or in non-empty elevator state; according to the at least one state identification result, determine the state of the escalator.

In a possible implementation manner, the determining the state of the escalator according to the at least one state identification result includes: in the case of multiple state identification results, determining the state of the escalator according to the multiple state identification results and The weights of the plurality of state identification results determine the state discriminating value of the escalator; when the state discriminating value is greater than or equal to the first threshold, it is determined that the escalator is in an empty state.

In a possible implementation manner, the state identification result includes a first state identification result, and the elevator state identification is performed on a first area image corresponding to the first area to determine at least one state of the escalator The identification result includes: classifying the first area image to obtain the first state identification result of the escalator.

In a possible implementation manner, the state identification result includes a second state identification result, and the elevator state identification is performed on the first area image corresponding to the first area to determine at least one state of the escalator The identification result includes: segmenting the first area image, dividing the first area image into a background area and a foreground area where the escalator is located; adjusting the pixel values of the background area to obtain an adjusted The second area image of the escalator is obtained; the second area image is classified and processed to obtain the second state recognition result of the escalator.

In a possible implementation manner, the state identification result includes a third state identification result, and the elevator state identification is performed on the first area image corresponding to the first area to determine at least one state of the escalator The identification result includes: performing pixel matching on the first area image and a preset reference image, and determining the matching area ratio between the first area image and the reference image, the reference image including The area image corresponding to the escalator in the elevator state; when the proportion of the matching area is greater than or equal to the second threshold, it is determined that the third state recognition result is that the escalator is in an empty state. In a possible implementation manner, the state identification result includes a fourth state identification result, and the elevator state identification is performed on the first area image corresponding to the first area to determine at least one state of the escalator The identification result includes: performing a first target detection on the first area image, and determining whether the first target exists in the first area image; and in the case where the first target does not exist in the first area image, determining The fourth state recognition result is that the escalator is in an empty state.

In a possible implementation manner, the method further includes: when the escalator is in an empty state, sending an elevator stop signal, where the elevator stop signal is used to instruct the escalator to stop running.

In a possible implementation manner, the method further includes: when the escalator is in a non-empty state and the escalator has stopped running, sending an elevator start signal, where the elevator start signal is used for Instruct the escalator to run.

In a possible implementation manner, the method further includes: performing second target detection on the first image, and determining a third area of the second target in the first image; The positional relationship between the third areas determines a detection result of the second target, where the detection result includes that the second target is on the escalator or not on the escalator. In a possible implementation manner, the determining the detection result of the second target according to the positional relationship between the first area and the third area includes: between the fourth area and the third area When the area ratio between the areas is greater than or equal to a third threshold, it is determined that the detection result is that the second target is on the escalator, and the fourth area includes the first area and the third Intersection area between regions.

In a possible implementation manner, the second target includes items prohibited from entering the escalator, and the method further includes: when the second target is on the escalator, sending alarm information. According to an aspect of the embodiments of the present disclosure, an image detection apparatus is provided, including: an image acquisition module configured to acquire a first image of an escalator; an area detection module configured to perform area detection on the first image, and determine the first area of the escalator in the first image; the state recognition module is configured to perform elevator state recognition on the first area image corresponding to the first area, and determine at least one state recognition of the escalator As a result, the state recognition result includes that the escalator is in an empty state or a non-empty state; the state determination module is configured to determine the state of the escalator according to the at least one state recognition result.

According to an aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to Perform the above method.

According to an aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.

According to an aspect of the embodiments of the present disclosure, there is provided a computer program product including computer-readable code, and when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method .

According to the embodiments of the present disclosure, it is possible to detect the area where the escalator is located in the image, identify at least one state recognition result according to the area image of the escalator, and determine the empty state or non-empty state of the escalator according to the at least one state recognition result The state can improve the positioning accuracy of the escalator area and improve the recognition accuracy of the running state of the escalator.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 shows a flowchart of an image detection method according to an embodiment of the present disclosure.

Fig. 2a shows a schematic diagram of region detection of an image detection method according to an embodiment of the present disclosure.

FIG. 2b shows a schematic diagram of area detection of an image detection method according to an embodiment of the present disclosure.

FIG. 3 shows a schematic diagram of a first region according to an embodiment of the present disclosure.

Figure 4a shows a schematic diagram of a second object according to an embodiment of the present disclosure.

Figure 4b shows a schematic diagram of a second object according to an embodiment of the present disclosure.

FIG. 5 shows a schematic diagram of a processing procedure of an image detection method according to an embodiment of the present disclosure.

FIG. 6 shows a block diagram of an image detection apparatus according to an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

FIG. 8 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

Implementation

Various exemplary embodiments, features and aspects of embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the present disclosure, numerous details are given in the following embodiments. It will be understood by those skilled in the art that the present disclosure may be practiced without certain details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

In recent years, the artificial intelligence wave based on deep learning, industry + artificial intelligence (AI, Artificial Intelligence) has become one of the goals of industrial development in the new era. New fields such as smart factories, smart stores, and smart agriculture are emerging one after another. Among them, the industrial application of AI for security, called smart security, is particularly popular. In the field of public safety, the escalator is a common means of transportation in modern life, and it is also a public place that needs to be managed by strict rules and regulations. The purpose of the smart elevator is to improve the work efficiency of the management department and strengthen the supervision of the daily operation of the elevator. In the related art, only part of the detection function for escalators can usually be realized, and there is no complete smart elevator solution.

The image detection method according to the embodiment of the present disclosure can be applied to scenes such as shopping malls, office buildings, and public transportation. The positioning of the elevator area, the discrimination of the empty and non-empty status of the elevator, and the detection and identification of key targets (such as baby strollers, wheelchairs, luggage, etc.) Effectively, reduce the cost of elevator operation and reduce the risk of safety accidents.

FIG. 1 shows a flowchart of an image detection method according to an embodiment of the present disclosure. As shown in FIG. 1 , the image detection method includes:

In step S11, a first image of the escalator is acquired;

In step S12, performing region detection on the first image to determine a first region of the escalator in the first image;

In step S13, perform elevator state recognition on the first area image corresponding to the first area, and determine at least one state recognition result of the escalator, where the state recognition result includes that the escalator is in an empty state or in a non-empty state;

In step S14, the state of the escalator is determined according to the at least one state identification result.

In a possible implementation manner, the image detection method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a handheld device, a computing equipment, vehicle-mounted equipment, wearable equipment, etc., the method can be implemented by the processor calling the computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server.

For example, at least one image acquisition device can be installed at the location where the escalator to be detected is located, for example, at least one camera facing the escalator can be installed, so as to collect the video stream of the escalator, and to monitor the escalator and the escalator in the video stream. The target of the escalator (such as pedestrians, items carried by pedestrians, etc.) is detected. The embodiments of the present disclosure do not limit the installation position of the image acquisition device, the acquisition method of the video stream, and the area corresponding to the video stream.

In a possible implementation manner, in step S11, a first image of the escalator may be acquired. Each image frame of the video stream can be used as the first image; the video stream can also be sampled at a certain time interval, and the sampled image frame can be used as the first image; key frames in the video stream can also be obtained as the first image . The embodiment of the present disclosure does not limit the acquisition manner of the first image.

In a possible implementation manner, in step S12, region detection may be performed on the first image through the trained region detection network to determine the first region of the escalator in the first image. The region detection network may be, for example, a convolutional neural network, and the embodiments of the present disclosure do not limit the network structure and training method of the region detection network.

2a and 2b illustrate schematic diagrams of region detection in an image detection method according to an embodiment of the present disclosure. Fig. 2a shows the first image of the escalator; Fig. 2b shows the region detection result in the first image.

In a possible implementation manner, the region detection network may adopt an object detection model or a region segmentation model. When the target detection model is used, the detected first area can be the rectangular detection frame of the area where the escalator is located, such as the detection frame 21 in Fig. 2b; when the area segmentation model is used, the detected first area can be the escalator The irregular region in which it is located, such as region 22 in Figure 2b. The embodiment of the present disclosure does not limit the network model adopted by the area detection network.

When the target detection model is used for processing, the background area can be reduced as much as possible while the detection frame includes the complete elevator area, thereby reducing the influence of background noise on the subsequent processing process. As shown in Figure 2b, when the elevator orientation is offset to the left, the lower left corner of the dark elevator area is used as the reference, as the coordinates of the lower left vertex of the detection frame 21; the upper right corner of the dark elevator area is used as the reference, as the detection The coordinates of the upper right vertex of box 21. When the region segmentation model is used for processing, the dark elevator region in the detection frame can be segmented, and a polygon can be drawn with the elevator handrail as the boundary, such as region 22 in FIG. 2b.

In a possible implementation manner, after the first area is obtained, in step S13, a first area image corresponding to the first area may be intercepted from the first image, and the escalator may be identified according to the first area image state, that is, to identify whether the escalator is in an empty state or a non-empty state. Among them, when there are no pedestrians and/or objects on the escalator, it is in an empty state; when there are pedestrians and/or objects on the escalator, it is in a non-empty state.

In a possible implementation manner, at least one identification manner may be used to identify the images of the first area respectively to obtain a corresponding state identification result. The identification method may include, for example, at least one of the following: directly classifying the image of the first area; adjusting the pixel value of the background area in the image of the first area, and classifying the adjusted image; comparing the image of the first area with the reference image of the empty elevator Comparison; detect whether there are objects such as pedestrians on the elevator. The embodiments of the present disclosure do not limit the identification methods used.

In a possible implementation manner, in step S14, the state of the escalator may be determined according to at least one state identification result. For example, referring to the voting mechanism, the state of the escalator is comprehensively determined according to the value and weight of each state identification result. When the state recognition result is that the overall weight of the empty elevator state is higher, it is determined that the escalator is in the empty elevator state; otherwise, it is determined that the escalator is in the non-empty elevator state.

In a possible implementation, when the escalator is in an empty state, an elevator stop signal can be sent to instruct the escalator to stop running; when the escalator is in a non-empty state and has stopped running, an elevator stop signal can be sent. Send an elevator start signal to instruct the escalator to start running. The embodiments of the present disclosure do not limit the operations performed in different elevator states.

According to the image detection method of the embodiment of the present disclosure, the area where the escalator is located in the image can be detected, at least one state recognition result is recognized according to the area image of the escalator, and the empty state of the escalator or the escalator is determined according to the at least one state recognition result. In the non-empty state, the positioning accuracy of the escalator area can be improved, and the recognition accuracy of the running state of the escalator can be improved.

The image detection method according to the embodiment of the present disclosure will be described below.

As mentioned above, the video stream of the area where the escalator is located can be captured by the camera, and the captured video stream can be transmitted to electronic devices such as a local front-end server or a cloud server. The electronic device can decode the video stream to obtain the decoded video stream.

In step S11, a first image of the escalator may be acquired. The first image may be an image frame of the decoded video stream. In step S12, the region detection is performed on the first image through the trained region detection network, and the first region of the escalator in the first image is determined, for example, the detection frame of the region where the escalator is located.

In step S13, the first area image corresponding to the first area may be intercepted from the first image, and at least one identification method is used to respectively identify the first area image to obtain at least one state identification result.

FIG. 3 shows a schematic diagram of a first region according to an embodiment of the present disclosure. As shown in FIG. 3 , the first image includes the

first areas

31 and 32 of two elevators. The first area images corresponding to the

first areas

31 and 32 can be intercepted respectively, and the two first area images can be recognized in parallel. Thereby improving the processing efficiency.

In a possible implementation manner, the state identification result includes the first state identification result. Step S13 may include:

The first area image is classified and processed to obtain a first state recognition result of the escalator.

That is, the first region image can be directly classified by the trained classification network (herein referred to as the first classification network). The first classification network may be, for example, a convolutional neural network, including a convolutional layer, a fully connected layer, an activation layer, etc. The embodiment of the present disclosure does not limit the network structure and training method of the first classification network.

In a possible implementation manner, the first region image can be input into the first classification network, and the first state recognition result can be output, indicating that the escalator is in an empty elevator state or in a non-empty elevator state, for example, when the escalator is in an empty elevator state It outputs 1 when it is in the state, and outputs 0 when it is in the non-empty state. In this way, the status of the escalator can be identified simply and efficiently.

In a possible implementation manner, the state identification result includes the second state identification result. Step S13 may include:

Segmenting the first area image, and dividing the first area image into a background area and a foreground area where the escalator is located;

Adjust the pixel value of the background area to obtain the adjusted second area image;

The second area image is classified and processed to obtain the second state recognition result of the escalator.

For example, in the case where the first area is a rectangular detection frame, the first area image corresponding to the first area can be segmented by the trained segmentation network, and the first area image is segmented into the background area and the motorized area. The foreground area where the escalator is located. The segmentation network may be, for example, a convolutional neural network, including a convolutional layer, a fully connected layer, an activation layer, etc. The embodiments of the present disclosure do not limit the network structure and training method of the segmentation network.

In a possible implementation manner, when the first area is an irregular area where the escalator is located, the area image corresponding to the enclosing rectangular frame of the first area may be used as the first area image. The first area can be directly used as the foreground area; the area other than the first area in the first area image is used as the background area to realize the segmentation of the first area image.

In a possible implementation manner, the pixel values of the pixels in the background area may be adjusted, for example, the pixel values of the background area are all adjusted to zero (black) to obtain an adjusted image of the second area. The pixel value of the background area may also be adjusted to other values, which is not limited in this embodiment of the present disclosure.

In a possible implementation manner, the images of the second region may be classified by a trained classification network (herein referred to as a second classification network). The second classification network may be, for example, a convolutional neural network, including a convolutional layer, a fully connected layer, an activation layer, etc. The second classification network may have the same network structure as the first classification network, but with different network parameters. The embodiments of the present disclosure do not limit the network structure and training method of the second classification network.

In a possible implementation, the second region image is input into the second classification network, and the second state recognition result is output, indicating that the escalator is in an empty state or in a non-empty state, for example, when the escalator is in an empty state It outputs 1 when it is in the empty state, and outputs 0 when it is in an empty state. In this way, in combination with image segmentation, the second region image after pixel adjustment is classified and processed, which can improve the accuracy of elevator state identification.

In a possible implementation manner, the state identification result includes a third state identification result. Step S13 may include:

Pixel matching is performed on the first area image and a preset reference image, and the matching area ratio between the first area image and the reference image is determined, and the reference image includes an escalator in an empty state. the corresponding area image;

When the proportion of the matching area is greater than or equal to the second threshold, it is determined that the third state recognition result is that the escalator is in an empty state.

For example, when the escalator is in an empty state, a single or multiple area images corresponding to the escalator can be acquired, for example, an image of the escalator is acquired by a camera, and an area image is obtained by performing area detection on the image.

In a possible implementation, if a single area image is acquired, the area image can be used as a reference image; if multiple area images are acquired, the multiple area images can be fused to obtain a reference image (or called the empty elevator template). For example, through a long-term image acquisition of 1 to 2 days, a segmentation model is used to obtain the labels of all pixels in the elevator area, and then an empty elevator template is obtained through a Gaussian mixture model. The embodiments of the present disclosure do not limit the generation manner of the reference image.

In one possible implementation, the reference images may be stored in a database. When performing elevator state recognition, pixel matching can be performed on the first area image and the reference image to determine the number of matching pixels; The proportion of matching areas.

In a possible implementation manner, if the proportion of the matching area is greater than or equal to the second threshold, it may be determined that the third state recognition result is that the escalator is in an empty state; otherwise, if the proportion of the matching area is less than the second threshold , it can be determined that the third state recognition result is that the escalator is in a non-empty state. Wherein, those skilled in the art can set the second threshold according to the actual situation, for example, 0.8, and the embodiment of the present disclosure does not limit the actual value of the second threshold.

Through pixel matching, the efficiency of elevator state recognition can be improved.

In a possible implementation manner, the state identification result includes a fourth state identification result. Step S13 may include:

performing a first target detection on the first area image to determine whether there is a first target in the first area image;

In the case where the first target does not exist in the first area image, it is determined that the fourth state recognition result is that the escalator is in an empty state.

For example, a trained target detection network (which may be referred to as a first detection network) may be used to perform a first target detection on an image of the first area to determine whether there is a first target in the image of the first area. The first target may include, for example, a pedestrian, an item, etc., which is not limited in this embodiment of the present disclosure.

In a possible implementation manner, the first detection network may be, for example, a convolutional neural network, and the embodiment of the present disclosure does not limit the network structure and training method of the first detection network.

In a possible implementation manner, if the first target exists in the first area image, it may be determined that the fourth state recognition result is that the escalator is in a non-empty state; otherwise, if the first area image does not exist the first target If a target is reached, it can be determined that the fourth state recognition result is that the escalator is in an empty state. By setting the first target and performing target detection on the image of the first area, the diversity of elevator state identification methods can be improved.

After each state recognition result is determined, the state of the escalator can be determined according to each state recognition result in step S14. Wherein, step S14 may include: in the case of multiple state identification results, determining the state identification value of the escalator according to the multiple state identification results and the weights of the multiple state identification results;

When the state discrimination value is greater than or equal to a first threshold value, it is determined that the escalator is in an empty state.

For example, if there is one state recognition result, the state of the escalator can be directly determined according to the state recognition result; if there are multiple state recognition results, the state of the escalator can be comprehensively determined according to the multiple state recognition results.

In a possible implementation manner, the weight of each state recognition result may be preset, the weight of the state recognition result with a higher accuracy rate is set to be higher, and the weight of the state recognition result with a lower accuracy rate is set to be higher low, the sum of the weights of each state recognition result is 1.

In a possible implementation manner, the state discrimination value of the escalator may be determined according to multiple state identification results and the weights of the multiple state identification results with reference to a voting mechanism. The state identification result may include at least two of the above-mentioned first, second, third and fourth state identification results.

For example, when the state recognition result includes the first state recognition result a ₁ , the second state recognition result a ₂ , the third state recognition result a ₃ , and the fourth state recognition result a ₄ , the first and second state recognition results , the weights of the third and fourth state identification results are set as w ₁ , w ₂ , w ₃ and w ₄ respectively, then the state discrimination value=w ₁ *a ₁ +w ₂ *a ₂ +w ₃ *a ₃ +w ₄ *a ₄ . Wherein, w ₁ +w ₂ +w ₃ +w ₄ =1, a ₁ , a ₂ , a ₃ , and a ₄ output 1 when the escalator is in an empty state, and output 0 when it is in a non-empty state.

In a possible implementation manner, if the state discriminating value is greater than or equal to the preset first threshold, it can be considered that the escalator is in an empty state; on the contrary, if the state discriminating value is smaller than the preset first threshold, it can be considered that the escalator is in an empty state; The escalator is not empty. As shown in FIG. 3 , the escalator in area 31 is in an empty state, and the escalator in area 32 is in a non-empty state.

Those skilled in the art can set the first threshold value, for example, 0.5, according to the actual situation, and the embodiment of the present disclosure does not limit the value of the first threshold value.

Through the above-mentioned voting mechanism, the running state of the escalator can be accurately identified, and the error rate of the elevator state judgment can be significantly reduced.

In a possible implementation manner, the image detection method according to the embodiment of the present disclosure may further include:

When the escalator is in an empty state, an elevator stop signal is sent, and the elevator stop signal is used to instruct the escalator to stop running.

That is, if it is determined in step S14 that the escalator is in an empty state, an elevator stop signal can be generated and sent to instruct the escalator to stop running. For example, the elevator stop signal is sent to the elevator control device, so that the elevator control device controls the escalator to stop running; the elevator stop signal can also be sent to the staff, so that the staff can control the escalator to stop running. The embodiment of the present disclosure does not limit the type and transmission manner of the elevator outage signal.

In this way, the elevator can be stopped when it is empty, thereby reducing the cost of elevator operation.

When the escalator is in a non-empty state and the escalator has stopped running, an elevator start signal is sent, and the elevator start signal is used to instruct the escalator to run.

That is to say, if it is determined in step S14 that the escalator is in a non-empty state and the escalator has stopped running, that is, there are pedestrians riding the escalator during the shutdown, an elevator start signal can be generated and sent to instruct the escalator to run . Similar to the above method, the elevator start signal can be sent to the elevator control device, so that the elevator control device controls the operation of the escalator; the elevator start signal can also be sent to the staff, so that the staff can control the operation of the escalator. The embodiment of the present disclosure does not limit the type and transmission mode of the elevator start signal.

In this way, the elevator can be started when someone takes the elevator and the elevator is out of service, thereby ensuring the normal use of the elevator.

performing second target detection on the first image, and determining a third area of the second target in the first image;

According to the positional relationship between the first area and the third area, a detection result of the second target is determined, and the detection result includes that the second target is on the escalator or not on the electric escalator on the escalator.

For example, after acquiring the first image of the escalator in step S11, a trained target detection network (which may be referred to as a second detection network) can also be used to perform second target detection on the first image to determine the first image. Whether there is a second target in . The second target may include items prohibited from entering the escalator, such as a baby stroller, a wheelchair, a large luggage, etc. The embodiment of the present disclosure does not limit the type of the second target.

In a possible implementation manner, the second detection network may be, for example, a convolutional neural network, and the embodiment of the present disclosure does not limit the network structure and training method of the second detection network.

In a possible implementation manner, if the second object exists in the first image, the second detection network may determine a region (which may be referred to as a third region) of the second object in the first image. 4a and 4b show schematic diagrams of a second object according to an embodiment of the present disclosure. The second object in FIG. 4 a is the stroller 41 , and the second object in FIG. 4 b is the luggage 42 . As shown in FIG. 4a and FIG. 4b, the detection frame where the second target is located, that is, the third area can be determined.

In a possible implementation manner, according to the first area of the escalator in the first image, the detection result of the second target may be determined through the positional relationship between the first area and the third area. That is, it is determined whether the second target is on the escalator according to the positional relationship.

In a possible implementation manner, the step of determining the detection result of the second target according to the positional relationship between the first area and the third area may include:

In the case that the area ratio between the fourth area and the third area is greater than or equal to a third threshold, it is determined that the detection result is that the second target is on the escalator, and the fourth area includes all The intersection area between the first area and the third area.

That is to say, the judgment can be made by the idea of obtaining the Intersection-over-Union (IOU). Assuming that the fourth area includes the intersection area between the first area and the third area, the area ratio between the fourth area and the third area can be obtained, that is, the area of the intersection area of the area where the second target is located and the elevator area is equal to the The ratio of the area of the area where the second target is located.

In a possible implementation manner, if the area ratio is greater than or equal to a preset third threshold, it may be determined that the detection result is that the second target is on the escalator; otherwise, if the area ratio is less than the preset third threshold , it can be determined that the detection result is that the second target is not on the escalator. Those skilled in the art can set the third threshold according to the actual situation, for example, 0.6, and the embodiment of the present disclosure does not limit the value of the third threshold.

In this way, the judgment of the area where the second target is located can be realized.

In a possible implementation manner, the second detection network may also directly perform second target detection on the first area image of the elevator area to determine whether there is a second target in the first area image. If the second object exists in the first area image, it may be determined that the second object is on the escalator. The embodiments of the present disclosure do not limit the adopted processing manner.

In a possible implementation manner, the second target includes items prohibited from entering the escalator, and the image detection method according to an embodiment of the present disclosure may further include: when the second target is on the escalator Next, send alarm information.

That is, for items prohibited from entering the escalator, if it is determined in the foregoing steps that the second target is on the escalator, alarm information can be sent to remind or directly control the escalator to stop running.

For example, the alarm information can be sent to the elevator monitoring equipment and/or staff in the monitoring room, so that the staff can control the escalator to stop running and/or go to the elevator for processing; the alarm information can also be sent to the elevator control equipment to make the elevator The control device controls the escalator to stop running. The embodiments of the present disclosure do not limit the type and sending manner of the alarm information.

In a possible implementation manner, an alarm may be issued in response to the second object appearing on the elevator for the first time, and the second object that appears in succession will repeat the alarm within a specified interval to avoid sending alarm information too frequently.

In this way, when items prohibited from entering the escalator can enter the elevator, an alarm can be issued in time, thereby reducing the risk of a safety accident.

FIG. 5 shows a schematic diagram of a processing procedure of an image detection method according to an embodiment of the present disclosure. As shown in Figure 5, in the processing process, the image of the escalator can be input in step 51 of image input; step 52 of elevator positioning and step 53 of key target detection are respectively performed on the image to improve the processing efficiency of the event; Step 52 of elevator positioning determines the elevator area 54 in the image; then, through step 55 of judging empty elevators, it is determined that the elevator is empty/non-empty 57, and then corresponding processing is performed.

In the example, the key target position 56 in the image is determined through the key target detection step 53, and the key target is the item that is prohibited from entering the elevator (violation item); according to the elevator area 54 and the key target position 56, it is determined whether the key target is on the elevator ; If the key target is on the elevator, alarm 58 of illegal items. In this way, the entire processing process of the image detection method according to the embodiment of the present disclosure can be realized.

In step 52 of elevator positioning, input the first image of the escalator as shown in Figure 2a, that is, the elevator, build a target detection or area segmentation model based on the deep learning algorithm, and output the elevator detection frame 21 as shown in Figure 2b, Or output the segmentation result of the region 22 as shown in Fig. 2b.

Among them, the target detection method needs to ensure that the detection frame includes the elevator area while reducing the background area as much as possible, so as to prevent background noise from interfering with subsequent elevator services. Therefore, the bounding box of the elevator as shown in Fig. 2b is defined as the detection box 21. At the same time, pay attention to the position of the elevator. Figure 2b is offset to the left, with the lower left corner of the elevator area as the benchmark (the lower left coordinate of the bounding box is determined from this), and the right boundary is based on the upper right corner of the elevator area as the benchmark. If the segmentation model is used to build a more accurate elevator area positioning module, the elevator area is defined as the elevator area within the detection frame 21, and a polygon is drawn with the elevator handrail as the boundary, such as area 22 in Figure 2b.

In step 55 of judging empty elevators, build a multi-channel parallel elevator state classification model based on deep learning, receive the processing results of the elevator positioning module (taking the detection results as an example), perform data processing simultaneously through various technical solutions, and finally make the results Fusion, determine the idle state of the elevator: an empty elevator or a non-empty elevator, as shown in FIG. 3 , the left elevator 31 is in an idle state, and the right elevator 32 is in a non-idle state. This process can be accomplished in several ways:

The first method is to extract the detection frame area, and output the elevator status through the classification network model;

Method 2: Extract the detection frame area, refer to the segmentation area, set the pixel value of the background area in the detection frame to 0 (black), and output the elevator status through the classification network model;

Mode 3, based on the template matching method, by matching the input with the empty elevator template in the database, and outputting the proportion of the matching area, if the matching proportion exceeds T (equivalent to the second threshold, T>=0.8), then confirm that the input elevator is Idle elevator, otherwise non-idle elevator. Among them, after 1 to 2 days of long-term data collection, the segmentation model is used to obtain the labels of all pixels in all elevator areas, and finally the template of the empty elevator (equivalent to the reference image) is obtained through the Gaussian mixture model, and the proportion of the matching area is obtained through the same The ratio of the number of pixels/total number of pixels is obtained;

Method 4: Based on the pedestrian and key target detection model, the elevator area is used as the input. If there is no target of interest in the elevator area, the output is an idle elevator, otherwise it is a non-empty elevator.

The final result is determined according to the voting mechanism, wherein the weights of the first to fourth methods are w ₁ , w ₂ , w ₃ and w ₄ , where w ₁ +w ₂ +w ₃ +w ₄ =1. Based on the above voting mechanism, the error rate of elevator category judgment can be greatly reduced. If the idle state gets a higher weight, the information notification mechanism is triggered.

In step S56 of the key target location, detection of key targets such as baby carriages, wheelchairs, and luggage is supported, and various event alarms are supported. The output results of the model are a detection target of a stroller 41 (equivalent to the second target) as shown in 4a, and a detection target of a suitcase 42 shown in 4b (equivalent to the second target). Finally, according to the processing result of step 52 of elevator positioning, it is judged whether the detection target exists in the elevator area, so as to give an alarm. The calculation method of whether the detection result exists in the elevator area is: intersection ratio=intersection area of detection target and elevator area/area of detection target, if the intersection ratio is greater than T (0<T<1, generally take 0.5 to 1), then Trigger the alarm mechanism. Through target tracking, a single target will only alarm when it appears, and the same target that appears in succession will be repeatedly alarmed within a specified interval.

According to the image detection method of the embodiment of the present disclosure, the entrance of the smart elevator solution can be constructed based on the positioning of the elevator area; in the elevator area, the multi-channel parallel empty elevator discrimination method and the result voting mechanism are used to determine whether the elevator is an empty elevator, Improve the accuracy of empty elevator prediction; through the empty elevator alarm, guide the management department to reduce the elevator running speed or even stop running, reduce energy consumption while ensuring the safety and convenience of travel; based on the target detection algorithm, locate the key targets in the elevator area, Give an alarm to the illegal objects, and improve the supervision of the management department.

According to the image detection method of the embodiment of the present disclosure, elevator area positioning can be performed based on a deep learning target detection/segmentation model, and multi-target detection is supported to locate multiple elevator areas at the same time. Moreover, the two area definition methods proposed by this method can reduce the interference of background noise while ensuring complete coverage of the elevator area. Among the smart elevator solutions in the related art, there is no solution involving elevator positioning at all.

According to the image detection method of the embodiment of the present disclosure, it is possible to judge whether the elevator is empty based on the multi-channel parallel empty elevator state discrimination method combined with the result voting mechanism, which can greatly reduce the error rate of output results, and also supports multi-area parallel classification. In the solution of the related art, the empty state of the elevator is usually judged by a single method, and the accuracy rate is low.

According to the image detection method of the embodiment of the present disclosure, by training a deep neural network to detect specific illegal items and identify all key targets within the field of view, the problem of inability to detect illegal items such as baby strollers, wheelchairs, and large suitcases in the related art is solved , and based on the elevator area and key target area, a specific IOU calculation method is used to judge whether the illegal items are on the elevator, which improves the accuracy of the judgment.

According to the image detection method of the embodiment of the present disclosure, an intelligent elevator system based on automatic positioning and key target detection is proposed, which can form a complete and stable intelligent elevator solution and can be applied to all current public elevator scenarios.

The method can be applied to smart cameras to locate elevator areas and identify empty elevators in elevator scenes, reduce energy consumption when the elevator is in an empty state, ensure that the elevators are empty before the elevator stops, and improve the work efficiency of the management department; Elevator area positioning and key target detection, alarm the illegal items on the elevator, and strengthen the supervision of the management department.

It can be understood that the foregoing method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the embodiments, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the embodiments of the present disclosure also provide image detection apparatuses, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any image detection method provided by the embodiments of the present disclosure, and the corresponding technical solutions and descriptions and refer to the methods part of the corresponding records.

FIG. 6 shows a block diagram of an image detection apparatus according to an embodiment of the present disclosure. As shown in FIG. 6 , the apparatus includes:

an image acquisition module 61, configured to acquire a first image of the escalator;

an area detection module 62, configured to perform area detection on the first image, and determine a first area of the escalator in the first image;

The state recognition module 63 is configured to perform elevator state recognition on the first area image corresponding to the first area, and determine at least one state recognition result of the escalator, where the state recognition result includes that the escalator is empty state or in a non-empty state;

The state determination module 64 is configured to determine the state of the escalator according to the at least one state identification result.

In a possible implementation manner, the state determination module includes: a discriminant value determination sub-module, configured to identify multiple states according to the multiple state identification results and the multiple state identification results when there are multiple state identification results The weight of the result determines the state discriminating value of the escalator; the state determination submodule is configured to determine that the escalator is in an empty state when the state discriminating value is greater than or equal to a first threshold.

In a possible implementation manner, the state identification result includes a first state identification result, and the state identification module includes: a first result determination sub-module, configured to perform classification processing on the first area image, and obtain the The first state recognition result of the escalator is described.

In a possible implementation manner, the state identification result includes a second state identification result, and the state identification module includes: a segmentation sub-module configured to segment the first region image, and divide the first region image into The image is divided into a background area and a foreground area where the escalator is located; the pixel adjustment sub-module is configured to adjust the pixel values of the background area to obtain an adjusted second area image; the second result determination sub-module is configured to In order to classify the image of the second area, a second state recognition result of the escalator is obtained.

In a possible implementation manner, the state identification result includes a third state identification result, and the state identification module includes: a pixel matching sub-module configured to perform pixel matching between the first region image and a preset reference image. matching, to determine the proportion of the matching area between the first area image and the reference image, where the reference image includes the area image corresponding to the escalator in the empty elevator state; the third result determination sub-module is configured to When the proportion of the matching area is greater than or equal to the second threshold, it is determined that the third state recognition result is that the escalator is in an empty state.

In a possible implementation manner, the state identification result includes a fourth state identification result, and the state identification module includes: a detection sub-module configured to perform a first target detection on the first area image, and determine the Whether the first target exists in the first area image; the fourth result determination submodule is configured to determine that the fourth state recognition result is the escalator when the first target does not exist in the first area image in an empty state.

In a possible implementation manner, the device further includes: a shutdown signal sending module configured to send an elevator shutdown signal when the escalator is in an empty state, and the elevator shutdown signal is configured as The escalator is instructed to stop running.

In a possible implementation manner, the device further includes: a start signal sending module configured to send an elevator start signal when the escalator is in a non-empty state and the escalator has stopped running, The elevator activation signal is configured to instruct the escalator to operate.

In a possible implementation manner, the apparatus further includes: a target detection module, configured to perform second target detection on the first image, and determine a third area of the second target in the first image; detecting A result determination module, configured to determine a detection result of the second target according to the positional relationship between the first area and the third area, where the detection result includes that the second target is on the escalator or not on the escalator.

In a possible implementation manner, the detection result determination module includes: a determination sub-module configured to determine, when the area ratio between the fourth region and the third region is greater than or equal to a third threshold, The detection result is that the second target is on the escalator, and the fourth area includes an intersection area between the first area and the third area.

In a possible implementation manner, the second target includes items that are prohibited from entering the escalator, and the apparatus further includes: an alarm information sending module configured to, when the second target is on the escalator, , send alarm information.

In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments, and the specific implementation may refer to the descriptions in the above method embodiments.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable code, when the computer-readable code is run on a device, a processor in the device executes a method configured to implement the image detection method provided by any of the above embodiments. instruction.

Embodiments of the present disclosure further provide another computer program product configured to store computer-readable instructions, which, when executed, cause the computer to perform the operations of the image detection method provided by any of the foregoing embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 7 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

7, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 and communication component 816 .

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method configured to operate on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM) , Static Random-Access Memory), Erasable Programmable Read-Only Memory (EPROM, Electrically Erasable Programmable Read-Only Memory), Programmable Read-Only Memory (PROM, Programmable Read-Only Memory), Read-Only Memory (ROM, Read Only Memory), magnetic memory, flash memory, magnetic disk or optical disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD, Liquid Crystal Display) and a touch panel (TP, Touch Panel). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC, Microphone) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker configured to output audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors configured to provide status assessment of various aspects of electronic device 800 . For example, the sensor component 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800, and the sensor component 814 can also detect the electronic device 800 or one of the electronic devices 800. Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a Complementary Metal-Oxide-Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi, Wireless Fidelity), a second-generation mobile communication technology (2G, The 2nd Generation) or a third-generation mobile communication technology (3G, The 3nd Generation) Generation), or their combination. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC, Near Field Communication) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID, Radio Frequency Identification) technology, Infrared Data Association (IrDA, Infrared Data Association) technology, Ultra Wide Band (UWB, Ultra Wide Band) technology, Bluetooth (BT, Blue Tooth) technology and other technologies to achieve.

In an exemplary embodiment, electronic device 800 may be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD) Processing Device), Programmable Logic Device (PLD, Programmable Logic Device), Field Programmable Gate Array (FPGA, Field Programmable Gate Array), controller, microcontroller, microprocessor or other electronic component implementation, configured to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 8 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 8, electronic device 1900 includes processing component 1922, which may include one or more processors, and a memory resource represented by memory 1932 configured to store instructions executable by processing component 1922, such as an application program. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server™), a graphical user interface based operating system (Mac OS X™) introduced by Apple, a multi-user multi-process computer operating system (Unix™). ), a free and open source Unix-like operating system (LinuxTM), an open source Unix-like operating system (FreeBSDTM) or similar systems.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as memory 1932 comprising computer program instructions executable by the processing component 1922 of the electronic device 1900 to accomplish the above-described method is also provided.

Embodiments of the present disclosure may be systems, methods and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer-readable storage media may include: portable computer disks, hard disks, random access memory (RAM, Random Access Memory), read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory, Portable Compact Disc Read-Only Memory (CD-ROM, Compact Disc Read-Only Memory), Digital Versatile Disc (DVD, Digital Video Disc), memory stick, floppy disk, mechanical coding device, such as a punch card on which instructions are stored Or the protruding structure in the groove, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, pseudocode, firmware instructions, state setting data, or in one or more Source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or it may be connected to an external computer (eg, using Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays, or programmable logic arrays, that can execute computer readable program instructions are personalized by utilizing state information of computer readable program instructions , thereby implementing various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability

In the embodiment of the present disclosure, by acquiring the first image of the escalator; performing area detection on the first image, the first area of the escalator in the first image is determined; The first area image of the escalator is used for elevator status recognition, and at least one status recognition result of the escalator is determined, and the status recognition result includes that the escalator is in an empty elevator state or in a non-empty elevator state; As a result, the state of the escalator is determined; in this way, the recognition accuracy of the running state of the escalator can be improved, and a complete and stable intelligent elevator solution can be formed, which can be applied to all current public elevator scenarios.

Claims

A shooting processing method, comprising:

Get the first image of the escalator;

Perform area detection on the first image to determine the first area of the escalator in the first image;

Perform elevator state recognition on the first area image corresponding to the first area, and determine at least one state recognition result of the escalator, where the state recognition result includes that the escalator is in an empty elevator state or in a non-empty elevator state ;

According to the at least one state identification result, the state of the escalator is determined.
The method according to claim 1, wherein the determining the state of the escalator according to the at least one state identification result comprises:

In the case of multiple state identification results, determining the state identification value of the escalator according to the multiple state identification results and the weights of the multiple state identification results;

When the state discrimination value is greater than or equal to a first threshold value, it is determined that the escalator is in an empty state.
The method according to claim 1 or 2, wherein the state identification result comprises a first state identification result,

The performing elevator state identification on the first area image corresponding to the first area to determine at least one state identification result of the escalator includes:

The first area image is classified and processed to obtain a first state recognition result of the escalator.
The method according to any one of claims 1 to 3, wherein the state identification result comprises a second state identification result,

The performing elevator state identification on the first area image corresponding to the first area to determine at least one state identification result of the escalator includes:

Segmenting the first area image, and dividing the first area image into a background area and a foreground area where the escalator is located;

Adjust the pixel value of the background area to obtain the adjusted second area image;

The second area image is classified and processed to obtain the second state recognition result of the escalator.
The method according to any one of claims 1 to 4, wherein the state identification result comprises a third state identification result,

The performing elevator state identification on the first area image corresponding to the first area to determine at least one state identification result of the escalator includes:

Pixel matching is performed on the first area image and a preset reference image, and the matching area ratio between the first area image and the reference image is determined, and the reference image includes an escalator in an empty state. the corresponding area image;

When the proportion of the matching area is greater than or equal to the second threshold, it is determined that the third state recognition result is that the escalator is in an empty state.
The method according to any one of claims 1 to 5, wherein the state identification result comprises a fourth state identification result,

The performing elevator state identification on the first area image corresponding to the first area to determine at least one state identification result of the escalator includes:

performing a first target detection on the first area image to determine whether there is a first target in the first area image;

In the case that the first target does not exist in the first area image, it is determined that the fourth state recognition result is that the escalator is in an empty state.
The method according to any one of claims 1 to 6, wherein the method further comprises:

When the escalator is in an empty state, an elevator stop signal is sent, and the elevator stop signal is used to instruct the escalator to stop running.
The method according to any one of claims 1 to 7, wherein the method further comprises:

When the escalator is in a non-empty state and the escalator has stopped running, an elevator start signal is sent, and the elevator start signal is used to instruct the escalator to run.
The method according to any one of claims 1 to 8, wherein the method further comprises:

performing second target detection on the first image, and determining a third area of the second target in the first image;

According to the positional relationship between the first area and the third area, a detection result of the second target is determined, and the detection result includes that the second target is on the escalator or not on the electric escalator on the escalator.
The method according to claim 9, wherein the determining the detection result of the second target according to the positional relationship between the first area and the third area comprises:

In the case that the area ratio between the fourth area and the third area is greater than or equal to a third threshold, it is determined that the detection result is that the second target is on the escalator, and the fourth area includes all The intersection area between the first area and the third area.
The method of claim 9 or 10, wherein the second target includes items prohibited from entering the escalator, the method further comprising:

When the second target is on the escalator, alarm information is sent.
An image detection device, comprising:

an image acquisition module, configured to acquire a first image of the escalator;

an area detection module, configured to perform area detection on the first image, and determine a first area of the escalator in the first image;

A state recognition module configured to perform elevator state recognition on a first area image corresponding to the first area, and determine at least one state recognition result of the escalator, where the state recognition result includes that the escalator is in an empty state or in a non-empty state;

The state determination module is configured to determine the state of the escalator according to the at least one state identification result.
The apparatus of claim 12, wherein the state determination module comprises:

a discriminant value determination submodule, configured to determine the state discriminant value of the escalator according to the multiple state identification results and the weights of the multiple state identification results when the state identification results are multiple;

The state determination submodule is configured to determine that the escalator is in an empty state when the state discrimination value is greater than or equal to a first threshold.
The apparatus according to claim 12 or 13, wherein the state identification result includes a first state identification result, and the state identification module includes:

The first result determination sub-module is configured to perform classification processing on the first area image to obtain a first state recognition result of the escalator.
The device according to any one of claims 12 to 14, wherein the state identification result includes a second state identification result, and the state identification module includes:

a segmentation submodule, configured to segment the first area image, and divide the first area image into a background area and a foreground area where the escalator is located;

a pixel adjustment sub-module, configured to adjust the pixel value of the background area to obtain an adjusted second area image;

The second result determination sub-module is configured to perform classification processing on the second area image to obtain a second state recognition result of the escalator.
The device according to any one of claims 12 to 15, wherein the state identification result includes a third state identification result, and the state identification module includes:

A pixel matching sub-module, configured to perform pixel matching on the first area image and a preset reference image, and determine the matching area ratio between the first area image and the reference image, the reference image including and The image of the area corresponding to the escalator in the empty state;

The third result determination sub-module is configured to determine that the third state identification result is that the escalator is in an empty state when the proportion of the matching area is greater than or equal to the second threshold.
The device according to any one of claims 12 to 16, wherein the state identification result includes a fourth state identification result, and the state identification module includes:

a detection submodule, configured to perform a first target detection on the first area image to determine whether there is a first target in the first area image;

The fourth result determination sub-module is configured to determine that the fourth state recognition result is that the escalator is in an empty state when the first target does not exist in the first area image.
The apparatus of any one of claims 12 to 17, wherein the apparatus further comprises:

A shutdown signal sending module is configured to send an elevator shutdown signal when the escalator is in an empty state, and the elevator shutdown signal is configured to instruct the escalator to stop running.
The apparatus of any one of claims 12 to 18, wherein the apparatus further comprises:

A start signal sending module is configured to send an elevator start signal when the escalator is in a non-empty state and the escalator has stopped running, and the elevator start signal is configured to instruct the escalator to run.
The apparatus of any one of claims 12 to 19, wherein the apparatus further comprises:

a target detection module, configured to perform second target detection on the first image, and determine a third area of the second target in the first image;

a detection result determination module, configured to determine a detection result of the second target according to the positional relationship between the first area and the third area, the detection result includes that the second target is in the escalator on or not on the escalator.
The apparatus according to claim 20, wherein the detection result determination module comprises:

A determination sub-module configured to determine that the detection result is that the second target is on the escalator when the area ratio between the fourth area and the third area is greater than or equal to a third threshold, so The fourth area includes an intersection area between the first area and the third area.
The apparatus of claim 20 or 21, wherein the second target includes items prohibited from entering the escalator, the apparatus further comprising:

The alarm information sending module is configured to send alarm information when the second target is on the escalator.
An electronic device comprising:

processor;

a memory configured to store processor executable instructions;

wherein the processor is configured to invoke instructions stored in the memory to perform the method of any one of claims 1-11.
A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the method of any one of claims 1 to 11 when executed by a processor.
A computer program comprising computer readable code, when the computer readable code is run in an electronic device, the execution of the processor in the electronic device is configured to implement the method of any one of claims 1 to 11 .