CN109271847B

CN109271847B - Abnormity detection method, device and equipment in unmanned settlement scene

Info

Publication number: CN109271847B
Application number: CN201810865617.8A
Authority: CN
Inventors: 侯章军; 杨旭东; 张晓博
Original assignee: Advanced New Technologies Co Ltd
Current assignee: Advanced New Technologies Co Ltd; Advantageous New Technologies Co Ltd
Priority date: 2018-08-01
Filing date: 2018-08-01
Publication date: 2023-04-07
Anticipated expiration: 2038-08-01
Also published as: US20210049370A1; TWI723411B; EP3779776A1; CN109271847A; TW202008320A; EP3779776B1; EP3779776A4; US11132559B2; SG11202010377RA; WO2020024691A1

Abstract

The embodiment of the specification provides an abnormal detection method, an abnormal detection device and abnormal detection equipment in an unmanned settlement scene, wherein a depth camera device with a shooting area at least comprising a detection area is arranged in the unmanned settlement scene, the human body object entering the detection area can be detected by using image data obtained from the depth camera device, the image data comprises a depth image and an RGB image, and the detection area is an area for detecting commodities to be settled before settlement, so that the purpose that whether the number of people in the detection area is abnormal or whether human body gestures are abnormal or not can be judged by using the depth image and the RGB image at least, and then whether a central control system is informed to stop commodity detection and trigger alarm operation or not is determined, and the loss of customers or merchants caused by abnormal conditions is avoided.

Description

Abnormity detection method, device and equipment in unmanned settlement scene

Technical Field

The present disclosure relates to the field of data processing, and in particular, to a method, an apparatus, and a device for detecting an anomaly in an unmanned settlement scenario.

Background

With the development of science and technology, unmanned settlement scenes are increasingly applied to daily life, such as unmanned stores, unmanned shopping malls and the like. After the customer purchases the commodity, the customer can enter the designated detection area for automatic detection and settlement. In the detection area, the central control system can automatically detect the commodities purchased by the customer and transmit commodity information to the payment platform to finish automatic settlement operation. However, in the detection area, an abnormal event may occur to cause a loss to the merchant or the customer, or the like. Therefore, there is a need to provide an efficient anomaly detection scheme in an unmanned settlement scenario.

Disclosure of Invention

To overcome the problems in the related art, the present specification provides a method, an apparatus, and a device for detecting an anomaly in an unmanned settlement scenario.

According to a first aspect of embodiments of the present specification, there is provided a method of detecting an abnormality in an unmanned settlement scene in which a depth imaging apparatus is provided, the imaging apparatus including at least a detection area which is an area where a commodity to be settled is detected before settlement, the method including:

acquiring image data in a depth camera, wherein the image data comprises a depth image and an RGB image;

detecting human body objects entering a detection area by using the image data, wherein the detection comprises one or more of people number detection and gesture detection,

when the preset abnormal conditions are determined to be met based on the detection result, the central control system is informed to stop commodity detection and trigger alarm operation, wherein the abnormal conditions comprise: the number of the human body objects entering the detection area is larger than a preset number threshold, and the gestures of the human body objects entering the detection area are one or more preset abnormal gestures.

In one embodiment, the step of detecting the number of people comprises:

acquiring a foreground image for representing a moving object from a depth image according to a background model and the currently acquired depth image, wherein the background model is obtained by performing background modeling by using the depth image acquired when no moving object passes through a detection area;

and if the foreground image is utilized to judge that the moving object is a human body object, analyzing the connected region of the foreground image by combining the depth value in the depth image, and obtaining the number of people in the detection region according to the analysis result.

In one embodiment, the depth image and the RGB image are obtained by a depth camera device capturing the same scene at the same time, and the gesture detection includes:

if the foreground image is used for judging that the moving object is a human body object, analyzing a connected region of the foreground image by combining a depth value in the depth image, and segmenting the human body region in the foreground image according to an analysis result to obtain a connected region representing an independent human body;

and determining the gesture of the human body object by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image.

In one embodiment, the determining the gesture of the human object by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image includes:

carrying out human hand positioning on a connected region representing an independent human body in the depth image;

if the hand region is obtained, performing hand skeleton detection on a region corresponding to the hand region in the RGB image, and determining the gesture of the human body object according to the detection result;

if the hand region is not obtained, obtaining the independent human body region corresponding to the RGB image according to the connected region representing the independent human body in the depth image, carrying out hand skeleton detection on the independent human body region in the RGB image, and determining the gesture of the human body object according to the detection result.

In one embodiment, the method further comprises one or more of:

when the depth image is used for detecting that the human body object enters the detection area of the unmanned body object, the central control system is informed to start commodity detection;

and when the depth image is used for detecting that the current human body object leaves and no other human body object enters the detection area, the central control system is informed to stop the commodity detection.

According to a second aspect of the embodiments of the present specification, there is provided an abnormality detection apparatus in an unmanned settlement scene in which a depth imaging device having an imaging area including at least a detection area that is an area where a commodity to be settled is detected before settlement is provided, the apparatus comprising:

a data acquisition module to: acquiring image data in a depth camera, wherein the image data comprises a depth image and an RGB image;

an anomaly detection module to: detecting human body objects entering a detection area by using the image data, wherein the detection comprises one or more of people number detection and gesture detection,

an exception handling module to: when the preset abnormal conditions are determined to be met based on the detection result, the central control system is informed to stop commodity detection and trigger alarm operation, wherein the abnormal conditions comprise: the number of the human body objects entering the detection area is larger than a preset number threshold, and the gestures of the human body objects entering the detection area are one or more preset abnormal gestures.

In one embodiment, the anomaly detection module is to:

In one embodiment, the depth image and the RGB image are obtained by a depth camera device capturing the same scene at the same time, and the anomaly detection module is configured to:

if the foreground image is used for judging that the moving object is a human body object, analyzing a connected region of the foreground image by combining a depth value in the depth image, and segmenting a human body region in the foreground image according to an analysis result to obtain a connected region representing an independent human body;

In one embodiment, the anomaly detection module is specifically configured to:

positioning the hands of the human body in a communication area representing the independent human body in the depth image;

According to a third aspect of the embodiments of the present specification, there is provided a computer apparatus including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein a depth imaging apparatus in which an imaging area at least includes a detection area is provided in an unmanned settlement scene, the detection area being an area where a commodity to be settled is detected before settlement, wherein the processor implements the following method when executing the program:

The technical scheme provided by the embodiment of the specification can have the following beneficial effects:

in the embodiment of the specification, the depth camera device with the shooting area at least comprising the detection area is arranged in the unmanned settlement scene, and the human body object entering the detection area can be detected by utilizing the image data obtained from the depth camera device, and the detection area is an area for detecting the commodity to be settled before settlement, so that whether the number of people in the detection area is abnormal or whether the human body gesture is abnormal can be judged by utilizing the depth image and the RGB image, and further whether the central control system is informed to stop commodity detection and trigger alarm operation is determined, and the loss of the customer or the merchant caused by the abnormal condition is avoided.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and together with the description, serve to explain the principles of the specification.

FIG. 1 is a flow chart illustrating a method for anomaly detection in an unmanned settlement scenario according to an exemplary embodiment of the present description.

FIG. 2 is a flow diagram illustrating another method for anomaly detection in an unattended settlement scenario according to an exemplary embodiment of the present description.

Fig. 3 is a schematic view of an application scenario of an anomaly detection method according to an exemplary embodiment of the present specification.

Fig. 4 is a hardware configuration diagram of a computer device in which an abnormality detection apparatus is located in an unmanned settlement scenario according to an exemplary embodiment of the present specification.

Fig. 5 is a block diagram of an anomaly detection apparatus in an unmanned settlement scenario, shown in accordance with an exemplary embodiment of the present specification.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the specification, as detailed in the appended claims.

The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present specification. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.

With the continuous deepening of mobile payment technology, when an unmanned supermarket appears, the traditional retail mode is completely overturned due to the characteristics of unattended operation and complete independent settlement of customers. After the customer purchases the commodity, the customer can enter the detection area to perform commodity detection, and then automatic settlement is realized, for example, the customer enters a 'payment gate' to perform settlement. However, due to the feature of "unattended operation", in the detection area, there may be an abnormal situation that the customer lifts the product to avoid the product being detected, or there may be an abnormal situation that the product purchased by other customers is recognized as the product purchased by the current customer.

In view of this, embodiments of the present disclosure provide an anomaly detection scheme in an unmanned settlement scene, where a depth camera device having a shooting area at least including a detection area is set in the unmanned settlement scene, and image data obtained from the depth camera device can be used to detect a human object entering the detection area, and since the image data includes a depth image and an RGB image, and the detection area is an area where a commodity to be settled is detected before settlement, it is possible to determine whether the number of people in the detection area is abnormal or whether a human gesture is abnormal by using the depth image and the RGB image, and further determine whether to notify a central control system to stop commodity detection and trigger an alarm operation, thereby avoiding a loss caused by an abnormal situation to a customer or a merchant.

The embodiments of the present specification are described below with reference to the accompanying drawings.

As shown in fig. 1, which is a flowchart illustrating an abnormality detection method in an unmanned settlement scene in which a depth imaging apparatus having an imaging area including at least a detection area is provided, the detection area being an area where a commodity to be settled is detected before settlement, according to an exemplary embodiment, the method may include:

in step 102, acquiring image data in a depth camera, wherein the image data comprises a depth image and an RGB image;

in step 104, detecting human body objects entering a detection area by using the image data, wherein the detection comprises one or more of people number detection and gesture detection,

in step 106, when it is determined that a preset abnormal condition is satisfied based on the detection result, the central control system is notified to stop the commodity detection and trigger an alarm operation, wherein the abnormal condition includes: the number of the human body objects entering the detection area is larger than a preset number threshold, and the gestures of the human body objects entering the detection area are one or more preset abnormal gestures.

In this embodiment, the unmanned settlement scene may be a scene in which no manual settlement of goods is performed, for example, a scene of an unmanned supermarket, an unmanned shop, an unmanned mall, or the like. In one embodiment, a customer can complete commodity detection operation in the detection area, and the central control system can send the detection result to the payment platform, so that the payment platform completes settlement operation. For example, entering the detection area indicated by the payment gate completes the detection of the article, and thus completes the settlement operation of the article. In one embodiment, the payment gate may be a payment gateway having at least one gate in which the merchandise detection may be performed and in which the merchandise settlement operation may or may not be performed. For example, in one implementation, the door is controlled to open and the customer is cleared after settlement is complete. In another implementation, since the payment account of the customer is associated in advance, the door can be controlled to be opened after the commodity detection is completed, and the settlement operation is subsequently executed without waiting for the settlement process to be finished. Thus, the payment channel may also be referred to as a detection channel. In one example, the payment channel may be closed. In yet another example, the payment channel may not be closed in order to enhance the user experience.

The method of the embodiment of the specification can be applied to an unmanned settlement scene. In one embodiment, the method can be applied to an embedded development board arranged in a payment channel, and the embedded development board can have a GPU operation function, so that the integration of anomaly detection and commodity detection is realized. Furthermore, a model compression mode can be adopted, and the calculation amount of the embedded development board is reduced.

In the unmanned settlement scene of the present embodiment, a depth imaging apparatus is provided in which the imaging area includes at least the detection area. The depth camera may be a camera capable of acquiring a depth image and an RGB image. Each pixel value of the depth image, which may also be referred to as a depth value or depth information for a point in the scene, may be used to characterize the distance of the point from the image capture device. Different depth imaging devices may employ different depth image acquisition methods. For example, a depth image may be obtained by a binocular matching method using a depth imaging apparatus composed of dual RGB cameras. For another example, a depth imaging device consisting of an RGB camera plus structured light head projector (infrared) + structured light depth sensor may be used to obtain a depth image using structured light detection. It should be understood that other depth imaging devices and other methods of obtaining a depth image may also be used, and are not described herein.

The shooting area of the depth camera device includes at least a detection area, and the depth camera device may be set at a position associated with the payment channel based on the purpose. In one embodiment, the depth camera device may be arranged at the end of the payment channel and arranged with the detection area available as a standard, so that the depth value in the depth image used for indicating the customer is smaller and smaller when the customer moves from the head end to the end of the payment channel.

In order to detect the human body object entering the detection area by using the image data, the detection area may be calibrated in advance to determine the relationship between the detection area and the image in the actual scene. In one embodiment, the detection region may be calibrated to correspond to a region in the image based on detecting a designated object disposed at a boundary of the detection region. For example, a specified object is set at the boundary of the detection area in the actual scene to define the boundary of the detection area. The method comprises the steps of acquiring a depth image and an RGB image from a depth camera device, performing image recognition by using the RGB image to detect a region of a specified object in the image, and acquiring a depth value of the specified object by using the depth image, so that automatic calibration of the detection region is completed. In one embodiment, the designated object may be a square plate to reduce the detection difficulty.

In the embodiment of the present specification, the human body object entering the detection area may be detected by using the image data to determine whether an abnormal event occurs, and when the abnormal event occurs, an alarm may be issued and the detection of the commodity may be stopped. The abnormal event can be determined according to a specific application scene, and different preset abnormal conditions are configured based on different abnormal events.

In one embodiment, to avoid mistakenly identifying the products of other customers as the products of the current customer, the number of people entering the detection area may be limited. The exception condition may include: the number of the human body objects entering the detection area is larger than a preset number threshold. Correspondingly, when the number of the human body objects entering the detection area is larger than a preset number threshold, the abnormal event is judged to occur. The preset number of people threshold is determined according to the equipment configured in the application scene. In one example, the preset number of people threshold may be 1, so as to limit the number of people entering the detection area to 1, and when the number of people is greater than 1, the commodity detection is stopped and an alarm is given.

Regarding the number of people detection, the number of people detection may be obtained based on detection of a depth image acquired from a depth imaging apparatus. For example, moving pedestrian detection, pedestrian segmentation are performed based on the depth image to obtain the number of people in the detection area. As an example of a specific implementation, the step of detecting the number of people may include:

The background modeling can be a dynamic and continuous updating process, and data can be continuously read and background modeling can be carried out again. For example, the background model obtained by the last update is adopted to judge whether a person exists in the detection area, an unmanned image is obtained at regular time, and the background model updated last time is updated by a Gaussian background modeling mode. Wherein the initial background model is obtained based on the initialization of the acquired unmanned image.

Comparing the currently acquired depth image (the depth image of the current frame) with the background model, a foreground image representing a moving object may be acquired from the depth image of the current frame. For example, comparing the depth image of the current frame with the background image, calculating the difference between the depth image and the background image at each corresponding pixel point, and finding out the position coordinates of each pixel point of which the difference value meets a certain condition, thereby obtaining the foreground image.

In the embodiment, the background modeling process can be completed through the depth video, when an object enters the detection area, the moving object with the change can be regarded as the moving foreground, and whether the moving object is a human body is judged through the image recognition technology.

Because the depth values of the same person are often relatively close and the same person has connectivity, the connected region analysis can be performed on the foreground image by combining the depth values in the depth image, and the number of people in the detection region can be obtained according to the analysis result. Connected Component (Connected Component) may refer to an image Region (Blob) composed of foreground pixels having similar pixel values and located adjacently in an image. Connected component analysis may refer to finding and labeling individual connected components in an image.

In one example, domain determination may be employed to perform connected component analysis on the foreground images. And judging whether each pixel point on the domain position of one pixel point is similar to the property of the pixel point or not, and grouping the pixel points with similar properties into a whole. In this embodiment, the property similarity may mean that the depth values are similar.

It can be seen from the above embodiments that, in the present embodiment, the foreground image is extracted by means of background modeling, and when it is determined that the moving object in the foreground image is a human body object, the connected region analysis is performed on the foreground image in combination with the depth value in the depth image, so as to implement human body segmentation.

In another embodiment, the customer may perform some abnormal behavior, particularly an abnormal gesture, in order to escape from the order, so that a gesture of a human body entering the detection area may be detected, and when the gesture of the human body object entering the detection area is a preset abnormal gesture, it is determined that an abnormal event occurs. The preset abnormal gesture may be determined according to a detectable range of the merchandise detection device, for example, a gesture capable of avoiding the detectable range of the merchandise detection device may be determined as the preset abnormal gesture. For example, the preset abnormal gesture may be a hand lifting gesture or a hand lifting gesture, so that the commodity can be prevented from being monitored when the customer performs the abnormal gesture. In some scenes, the abnormal gesture can be distinguished according to the left hand and the right hand, so that different judgment results are different when different hands execute the same gesture. The abnormal gestures of the left hand and the right hand can depend on specific scenes.

Regarding gesture detection, gesture detection may be obtained based on detection of a depth image and an RGB image acquired from a depth camera device. The depth image and the RGB image are obtained by acquiring the same scene at the same time by the depth camera equipment. The depth image and the RGB image are images reflecting the same scene by different data, and the depth image reflects the distance from a certain point in the scene to the camera equipment by adopting a depth value. In one example, based on the depth image and the RGB image, moving pedestrian detection, pedestrian segmentation, human hand positioning and capturing, and gesture recognition are performed to obtain the gesture of the pedestrian within the detection area. As illustrated below in one specific implementation, the step of gesture detection may include:

The foreground image acquisition, the human body object judgment, the connected region analysis, and the like are the same as those of the people detection, and are not repeated herein. In the embodiment, the depth image is adopted to obtain the connected region representing the independent human body, and compared with the method for obtaining the connected region representing the independent human body by adopting the RGB image, the method can save the operation amount.

In this embodiment, after the connected region representing the independent human body is obtained through the depth image, the gesture of the human body object may be determined by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image. After the connected region is obtained, gesture recognition can be performed by combining the RGB image and the depth image. For example, the confidence level of the hand region is determined by using the depth image, and if the confidence level is greater than a preset threshold value, the hand region in the depth image is mapped into the RGB image, and the gesture determination is performed on the region corresponding to the hand region in the RGB image. If the hand region cannot be judged through the depth image, the connected region in the depth image is mapped to the RGB image, and the hand region and the gesture are judged through the RGB image. Specifically, the determining the gesture of the human body object by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image may include:

In this embodiment, an individual depth connected region of each human body may be acquired from information of human body segmentation, a hand region with respect to a front end of the human body may be acquired from depth information of the individual human body, and finally, left and right hand determination and extraction may be performed from relative positions of the front end region and the body region, thereby obtaining the hand region. And performing hand interception in a corresponding area in the RGB image by using the position of the hand area acquired in the depth image to obtain a hand image. And performing hand skeleton shape recognition on the hand image to acquire the hand skeleton shape, and performing gesture recognition on the basis. For example, the skeleton of the hand can be acquired by a hand skeleton detection method, and then the hand motion can be recognized by shape discrimination of five finger skeletons in the hand skeleton.

According to the embodiment, the hand region is obtained by utilizing the depth image, the hand skeleton detection is carried out on the region corresponding to the hand region in the RGB image, the gesture of the human body object is determined according to the detection result, and the calculation amount can be reduced.

It should be understood that the abnormal condition may further include other conditions, for example, the human body posture of the human body object entering the detection area is a preset abnormal posture, and therefore, the detection may further include human body posture detection, which is not described herein for other situations.

When the preset abnormal condition is met, the central control system can be informed to stop the commodity detection and trigger the alarm operation. Commodity detection is a preceding work of the settlement operation, and therefore, when commodity detection is interrupted, the settlement operation is also interrupted. The central control system may be a system for detecting the commodity before settlement. The detection of the commodity may include identifying the commodity, and obtaining commodity information such as the quantity of the commodity, the price, and the like required for settlement. For example, radio Frequency Identification (RFID) technology may be used to read the electronic tags on the goods to obtain the goods information required for settlement. Of course, other means may be adopted to detect the commodity to obtain the commodity information required for settlement, which is not limited herein. The alarm operation can be triggered by the execution end through the central control system, and the execution end can also directly inform the alarm module of the alarm operation.

As can be seen from the above embodiments, in this embodiment, by setting the depth camera in which the shooting area at least includes the detection area in the unmanned settlement scene, the human body object entering the detection area can be detected by using the image data acquired from the depth camera, and the detection area is an area for detecting the commodity to be settled before settlement, so that it is possible to determine whether the number of people in the detection area is abnormal by using the depth image, or determine whether the human body gesture in the detection area is abnormal by using the depth image and the RGB image, and then determine whether to notify the central control system to stop the commodity detection and trigger the alarm operation, thereby avoiding the loss of the abnormal situation to the customer or the merchant.

In order to avoid the waste of resources caused by real-time detection of the central control system, in one embodiment, when the depth image is used for detecting that a human body object enters the detection area of an unmanned body object, the central control system is notified to start commodity detection.

In this embodiment, the merchandise detection function may be turned on when the detection area is from an unmanned object to a manned object. Therefore, when a human body object enters the detection area of the unmanned object, the commodity detection operation is triggered, and the commodity detection function can be automatically started.

In another embodiment, when it is detected by the depth image that the current human body object leaves and no other human body object enters the detection area, the central control system is notified to stop the commodity detection.

In this embodiment, when a human subject leaves the detection area and no other person enters the detection area, the merchandise detection function may be turned off. Further, if the current human body object leaves the detection area within a preset time period, if no other human body object enters the detection area, the central control system can be notified to stop the commodity detection.

Therefore, the commodity detection function can be automatically closed through the depth image, and resource waste caused by real-time detection of a central control system is avoided.

Any combination of the technical features in the above embodiments may be arbitrarily combined as long as there is no conflict or contradiction in the combination of the features, but the combination is limited by space and is not described one by one, so that any combination of the technical features in the above embodiments also falls within the scope disclosed in the present specification.

One of the combinations is exemplified below.

As shown in fig. 2, a flowchart of another method for detecting an abnormality in an unattended settlement scene in which a depth imaging apparatus having an imaging area including at least a detection area is provided, the detection area being an area for detecting a commodity to be settled before settlement, according to an exemplary embodiment, is shown in the present specification, and the method includes:

in step 202, the depth imaging apparatus is acquired to acquire a depth image and an RGB image of the same scene at the same time.

The depth image pickup device can be in a normally open state, or in a normally open state in a specified working time period. For example, the designated work hours may be business hours of an unmanned store.

In step 204, moving pedestrian detection is carried out by using the depth image, and if a human body object is detected to enter a detection area without the human body object, a central control system is informed to start commodity detection; and if the current human body object is detected to leave and no other human body object enters the detection area, the central control system is informed to stop the commodity detection.

In one example, a foreground image representing a moving object may be obtained from the depth image according to the background model and the currently obtained depth image, and it may be determined whether the moving object is a human body object using the foreground image.

In step 206, the pedestrian detection result is segmented to obtain a connected region representing an independent human body and the number of people of the human body object entering the detection region, and the gesture of the human body object is determined by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image.

The foreground image can be subjected to connected region analysis by combining with the depth value in the depth image so as to perform pedestrian segmentation.

In step 208, if the number of human body objects entering the detection area is greater than the preset number threshold, the central control system is notified to stop the commodity detection and trigger the alarm operation.

In step 210, if the gesture of the human body object entering the detection area is a preset abnormal gesture, the central control system is notified to stop the commodity detection, and an alarm operation is triggered.

Fig. 2 is the same as the related art in fig. 1, and is not repeated herein.

In the embodiment, moving pedestrian detection and pedestrian segmentation are realized through the depth image, and the gesture of the human body object is determined by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image, so that the calculation amount can be reduced; and can automatic control commodity detect opening and closing of function, avoid the commodity to detect the wasting of resources that the function leads to in the normally open state, can inform central control system to stop commodity detection and trigger the alarm operation when the number of people is unusual and human gesture is unusual in the detection area moreover, avoid the loss that the abnormal conditions caused for customer or trade company.

The embodiment of the present specification further exemplifies an abnormal detection scheme in an unmanned settlement scene by taking a specific application scene as an example. Fig. 3 is a schematic diagram illustrating an application scenario of an anomaly detection method according to an exemplary embodiment of the present specification. In this embodiment, the payment channel may be non-enclosed, and the depth camera may be an RGBD camera disposed at an end of the payment channel. The area where the payment channel is located is a detection area. Because the depth image and the RGB image can be obtained from the RGBD camera in real time, whether a customer walks into the detection area can be judged by using the depth image, and if yes, the central control system can be informed to start the commodity detection function. The depth image and the RGB image are used for judging whether an abnormal event occurs, when the abnormal event occurs, the central control system is prevented from detecting commodities held by customers, and an alarm operation is triggered, so that benefit conflict caused by the abnormal event is avoided. If no abnormal event occurs, the central control system can continue to detect the commodity, transmit the obtained commodity information to the payment platform and complete settlement operation by using the payment platform. For example, when a plurality of customers exist in the detection area, the central control system may be controlled to stop detecting the commodities held by the customers and trigger an alarm operation. When a customer executes an abnormal gesture in the detection area, the central control system can be controlled to stop detecting commodities held by the customer and trigger an alarm operation. The tail end of the payment channel can be provided with a valve, and when the detection is completed or the settlement is completed, the valve is opened to release customers.

Corresponding to the foregoing embodiments of the anomaly detection method in an unattended settlement scene, the present specification also provides embodiments of an anomaly detection apparatus in an unattended settlement scene and an electronic device applied thereto.

The embodiment of the anomaly detection device in the unmanned settlement scene in the specification can be applied to computer equipment, and the computer equipment can have a GPU operation function. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. The software implementation is taken as an example, and is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for running through the processor of the computer device where the software implementation is located as a logical means. From a hardware level, as shown in fig. 4, the hardware structure diagram of the computer device where the anomaly detection apparatus is located in the unmanned settlement scenario in this specification is shown, except for the processor 410, the network interface 420, the memory 430, and the nonvolatile memory 440 shown in fig. 4, in the embodiment, the computer device where the anomaly detection apparatus 431 is located in the unmanned settlement scenario may further include other hardware according to the actual function of the device, which is not described again.

As shown in fig. 5, the present specification is a block diagram of an abnormality detection apparatus in an unmanned settlement scene in which a depth imaging device having an imaging area including at least a detection area is provided, the detection area being an area where a commodity to be settled is detected before settlement, according to an exemplary embodiment, the apparatus including:

a data acquisition module 52 to: acquiring image data in a depth camera, wherein the image data can comprise a depth image and an RGB image;

an anomaly detection module 54 to: detecting human body objects entering a detection area by using the image data, wherein the detection comprises one or more of people number detection and gesture detection,

an exception handling module 56 for: when the preset abnormal conditions are determined to be met based on the detection result, the central control system is informed to stop commodity detection and trigger alarm operation, wherein the abnormal conditions comprise: the number of the human body objects entering the detection area is larger than a preset number threshold, and the gestures of the human body objects entering the detection area are one or more preset abnormal gestures.

In one embodiment, the anomaly detection module 54 is configured to:

In one embodiment, the depth image and the RGB image are obtained by a depth camera device capturing the same scene at the same time, and the anomaly detection module 54 is configured to:

In one embodiment, the anomaly detection module 54 is specifically configured to:

In one embodiment, the exception handling module 56 is further configured to perform one or more of the following:

For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in the specification. One of ordinary skill in the art can understand and implement without inventive effort.

Accordingly, an embodiment of the present specification further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a depth image capturing device is provided in an unmanned settlement scene, wherein a shooting area at least includes a detection area, and the detection area is an area for detecting a commodity to be settled before settlement, and the processor implements the following method when executing the program:

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

A computer storage medium having stored therein program instructions, the program instructions comprising:

Embodiments of the present description may take the form of a computer program product embodied on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having program code embodied therein. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of the storage medium of the computer include, but are not limited to: phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, may be used to store information that may be accessed by a computing device.

Other embodiments of the present description will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the specification following, in general, the principles of the specification and including such departures from the present disclosure as come within known or customary practice within the art to which the specification pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the specification being indicated by the following claims.

It will be understood that the present description is not limited to the precise arrangements described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims

1. The method is suitable for an embedded development board arranged in a payment channel, a depth camera is arranged in the unmanned settlement scene, the shooting area of the depth camera at least comprises a detection area, and the detection area is an area for detecting commodities to be settled before settlement, and the method comprises the following steps:

detecting a human body object entering a detection area by using the image data, wherein the detection comprises gesture detection, and the gesture detection comprises: performing connected region analysis on the depth image to obtain a hand region, mapping the hand region to the RGB image, determining a region corresponding to the hand region in the RGB image, performing hand skeleton detection on the corresponding region, and performing gesture recognition according to a detection result;

when the preset abnormal conditions are determined to be met based on the detection result, the central control system is informed to stop commodity detection and trigger alarm operation, wherein the abnormal conditions comprise: the gesture of the human body object entering the detection area is a preset abnormal gesture, and the abnormal gesture is a gesture for preventing the commodity to be settled from being detected.

2. The method of claim 1, the detecting further comprising a person number detection, the abnormal condition further comprising: the number of people of the human body object entering the detection area is larger than a preset number threshold.

3. The method of claim 2, the step of people number detection comprising:

4. The method of claim 1, wherein the depth image and the RGB image are obtained by a depth camera device capturing the same scene at the same time, and the gesture detection comprises:

5. The method of claim 4, wherein determining the gesture of the human object by combining the connected region in the depth image and the region corresponding to the connected region in the RGB image comprises:

if the hand area is obtained, performing hand skeleton detection on an area corresponding to the hand area in the RGB image, and determining the gesture of the human body object according to the detection result;

6. The method of any one of claims 1 to 5, further comprising one or more of:

7. An abnormality detection device in an unmanned settlement scene, the device being adapted to an embedded development board provided in a payment channel, a depth camera being provided in the unmanned settlement scene, a shooting area of the depth camera at least including a detection area, the detection area being an area where goods to be settled are detected before settlement, the device comprising:

an anomaly detection module to: detecting a human object entering a detection area using the image data, the detecting including gesture detection, the gesture detection including: performing connected region analysis on the depth image to obtain a hand region, mapping the hand region to the RGB image, determining a region corresponding to the hand region in the RGB image, performing hand skeleton detection on the corresponding region, and performing gesture recognition according to a detection result;

an exception handling module to: when the preset abnormal conditions are determined to be met based on the detection result, the central control system is informed to stop commodity detection and trigger alarm operation, wherein the abnormal conditions comprise: the gesture of the human body object entering the detection area is a preset abnormal gesture, and the abnormal gesture is a gesture for preventing the commodity to be settled from being detected.

8. The apparatus of claim 7, the detection further comprising a number of people detection, the exception condition further comprising: the number of the human body objects entering the detection area is larger than a preset number threshold.

9. The apparatus of claim 8, the anomaly detection module to:

10. The apparatus of claim 7, wherein the depth image and the RGB image are obtained by a depth camera device capturing the same scene at the same time, and the anomaly detection module is configured to:

11. The apparatus of claim 10, the anomaly detection module to specifically:

12. The apparatus of any of claims 7 to 11, the exception handling module further to perform one or more of:

13. A computer device, comprising a memory, a processor and a computer program stored on the memory and operable on the processor, wherein a depth camera device is provided in an unmanned settlement scene, the shooting area of the depth camera device at least comprises a detection area, and the detection area is an area for detecting a commodity to be settled before settlement, wherein the processor implements the following method when executing the program:

detecting a human body object entering a detection area by using the image data, wherein the detection comprises gesture detection, and the gesture detection comprises: analyzing a connected region of the depth image to obtain a hand region, mapping the hand region to an RGB image, determining a region corresponding to the hand region in the RGB image, detecting a hand skeleton of the corresponding region, and recognizing a gesture according to a detection result;

when determining that preset abnormal conditions are met based on the detection result, informing a central control system to stop commodity detection and triggering alarm operation, wherein the abnormal conditions comprise: the gesture of the human body object entering the detection area is a preset abnormal gesture, and the abnormal gesture is a gesture for preventing the commodity to be settled from being detected.