WO2020134101A1

WO2020134101A1 - Sales apparatus and sales method

Info

Publication number: WO2020134101A1
Application number: PCT/CN2019/099761
Authority: WO
Inventors: 刘自银; 刘朋樟; 张屹峰; 陈宇; 刘巍; 安山; 姜博; 乌日尼; 郭景昊; 张夏杰
Original assignee: 北京沃东天骏信息技术有限公司
Priority date: 2018-12-29
Filing date: 2019-08-08
Publication date: 2020-07-02
Also published as: CN109389747A

Abstract

A sales apparatus and a sales method. A sales apparatus (10) comprises: a processor (11), and a shopping area intrusion detection sensor (12) and a boundary area through detection sensor (13) which are electrically connected to the processor (11). The processor (11) is configured to determine, on the basis of detection information of at least one of the shopping area intrusion detection sensor (12) and the boundary area through detection sensor (13), the occurrence of a predetermined event, and trigger an operation associated with the predetermined event. At least one problem of abnormal settlement, a large amount of calculation for item identification, etc. comprised in the sales service can be solved.

Description

Vending device and method

cross reference

This application is based on the application with the CN application number 201811628559.3 and the application date is December 29, 2018, and claims its priority. The disclosure content of the CN application is hereby incorporated into this application as a whole.

Technical field

The present disclosure relates to the field of computers, and in particular, to a vending device and a vending method.

Background technique

With the development of computer technology, vending has been realized. People can take the items from the vending equipment, and can recognize the taken items based on the gravity of the items or the image of the items, and settle the items.

Summary of the invention

Some embodiments of the present disclosure provide a vending device, including:

processor,

as well as

At least one of the shopping area intrusion detection sensor and the boundary area passing through the detection sensor,

Wherein, the processor is electrically connected to the shopping area intrusion detection sensor, and the processor is electrically connected to the boundary area through the detection sensor;

The shopping area intrusion detection sensor is configured to detect whether the shopping area is invaded;

The boundary area passing through the detection sensor is configured to detect whether the boundary area between the placement area of the item to be sold and other areas is crossed;

The processor is configured to determine the occurrence of a predetermined event based on the detection information of at least one of the shopping area intrusion detection sensor and the boundary area penetration detection sensor, and trigger an operation associated with the predetermined event.

In some embodiments, the processor is configured to determine the occurrence of an event that the user enters or leaves the shopping area based on the detection information of the shopping area intrusion detection sensor.

In some embodiments, the processor is configured to determine the occurrence of an event in which the boundary area is traversed or an event in which the boundary area is traversed based on the detection information of the boundary area through the detection sensor.

In some embodiments, the association relationship between the predetermined event and the operation is configured as at least one of the following relationships:

The event that the user enters the shopping area is related to the operation of item identification or the operation of associating the cached sensing information used for item identification;

The event that the user leaves the shopping area is related to the operation of item settlement;

The event that the boundary area is crossed is related to the operation of identifying the item or the operation of adding or deleting the related purchased item;

The demarcation area is associated with the operation of identifying the item or the operation of adding or deleting the associated purchased item by the event of the crossing direction.

In some embodiments, the event that the user enters the shopping area is associated with an operation of item recognition based on at least one of item gravity and item image.

In some embodiments, the event that the demarcation area is traversed is associated with an operation of item recognition based on at least one of item gravity and item image, or associated with a purchased item implemented with at least one item gravity and item image Add or delete operations.

In some embodiments, the event that the demarcation area is traversed in a direction, associates an operation of item recognition based on at least one of item gravity and item image, or associates a purchase that combines item gravity and at least one item image Add or delete items.

In some embodiments, the processor is configured to retrieve an image of an item within a preset range based on the occurrence time of the event where the boundary area is traversed or the occurrence time of the event where the boundary area is traversed, Used for item identification or addition or deletion of purchased items.

In some embodiments, the processor is configured to determine the occurrence of an abnormal event based on detection information of at least one of the shopping area intrusion detection sensor and the boundary area passing detection sensor and the shopping information detection sensor, and Trigger the operation associated with the abnormal event,

Wherein, the shopping information detection sensor is configured to detect the item information purchased by the user from the vending device where the item to be sold is placed.

In some embodiments, the boundary area is set according to a plane with an angle smaller than a preset angle with the front of the vending device where the items to be sold are placed, and the distance between the boundary area and the front of the vending device is less than the preset distance.

In some embodiments, the shopping area is set according to a preset area area where the vending equipment is located.

In some embodiments, the shopping area intrusion detection sensor is a first image sensor, a photoelectric sensor in reflective working mode, an ultrasonic distance sensor, or a human body induction switch, wherein the detection area of the first image sensor is aimed at shopping area.

In some embodiments, the boundary area passing detection sensor is a second image sensor, or a photoelectric sensor in a contra-beam operating mode, wherein the detection area of the second image sensor is aligned with the boundary area.

Some embodiments of the present disclosure propose a sales method, which is applied to a sales device, including:

Obtain the detection information of whether the shopping area is invaded;

Obtain the detection information of whether the boundary area between the area where the item to be sold is placed and other areas is crossed;

Determine the occurrence of a predetermined event based on at least one detection information; and

The operation associated with the predetermined event is triggered in order to realize sales.

In some embodiments, based on the detection information of whether the shopping area is invaded, it is determined that the user enters or leaves the shopping area.

In some embodiments, in the event that the user enters the shopping area, the operation of buffering the sensing information used for item identification is triggered;

In the event that the user leaves the shopping area, an item settlement operation is triggered.

In some embodiments, based on the detection information of whether the boundary area between the placement area of the item to be sold and other areas is traversed, the occurrence of the event that the boundary area is traversed or the direction in which the boundary area is traversed is determined.

In some embodiments, in the event that an event that the boundary area is traversed occurs, the operation of item identification is triggered;

In the event that an event occurs in the direction in which the boundary area is crossed, trigger the operation of item identification, and the operation of adding or deleting purchased items.

In some embodiments, the operation of determining the removal or replacement of the item in combination with at least one of the gravity of the item and the image of the item triggers the addition or deletion of the purchased item based on the determined operation of removing or returning the item.

In some embodiments, based on the occurrence time of the event where the boundary area is traversed or the occurrence time of the event where the boundary area is traversed, the item image within the preset range is retrieved, based on the retrieved item image or a combination Item gravity for item identification.

Some embodiments of the present disclosure propose a non-transitory computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the sales method of any embodiment.

BRIEF DESCRIPTION

The drawings needed to be used in the embodiments or related technical descriptions will be briefly introduced below. The present disclosure can be more clearly understood from the following detailed description with reference to the drawings.

Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without paying any creative labor, other drawings can also be obtained based on these drawings.

FIG. 1 shows a schematic diagram of some embodiments of the vending device 10 of the present disclosure.

2 shows a schematic diagram of some examples of the shopping area 21 and the boundary area 22 relative to the vending device 23 of the present disclosure.

FIG. 3 shows a schematic diagram of the detection principle of the photoelectric sensor 30 in the reflective working mode according to some embodiments of the present disclosure.

FIG. 4 shows a schematic diagram of the detection principle of the ultrasonic ranging sensor 40 of some embodiments of the present disclosure.

FIG. 5 shows a schematic diagram of the detection principle of the human body induction switch 50 of some embodiments of the present disclosure.

FIG. 6 shows a schematic diagram of the detection principle of the photoelectric sensor 60 in an epi-working mode according to some embodiments of the present disclosure.

FIG. 7 shows a schematic flowchart of some embodiments of the sales method of the present disclosure.

FIG. 8 shows a schematic flowchart of some embodiments of the sales method of the present disclosure.

detailed description

The technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the drawings in the embodiments of the present disclosure.

As shown in FIG. 1, the sales apparatus 10 of this embodiment includes a processor 11, and various sensors electrically connected to the processor 11, for example, a shopping area intrusion detection sensor 12, a boundary area passing through the detection sensor 13, shopping The information detection sensor 14 and the like are not limited to the examples.

Items for sale can be placed on vending equipment. The vending equipment can be, for example, an open shelf or a closed container. These sensors can be installed on vending equipment. The sensor can be installed in the appropriate position of the sales equipment according to the different functions. The processor 11 may be integrated in the vending equipment.

The shopping area refers to an area where there is a high probability of shopping behavior occurring near the vending equipment and can be set in advance. For example, the shopping area is set according to a preset area area where the sales equipment is located.

The boundary between the area where the items for sale are placed and other areas is easy to understand. Taking the boundary area as the boundary, items for sale are placed on one side of the boundary area, while items for sale are not placed on the other side. The boundary area can be set according to a plane whose angle with the front of the vending equipment is less than the preset angle, and the distance between the boundary area and the front of the vending equipment is less than the preset distance. If the external contour of the vending device is or is approximately a rectangular parallelepiped, the outer surface of the vending device facing the user side (ie, the side where the user can take the goods to be sold) is regarded as the front of the vending device. For example, the boundary area may be set as a plane perpendicular to the ground and parallel to the front of the vending equipment, or the boundary area may be set as the front of the vending equipment.

2 shows a schematic diagram of some examples of the shopping area 21 and the boundary area 22 relative to the vending device 23 of the present disclosure. In some embodiments, the demarcation area may be set as an area closer to the vending device than the outer edge of the shopping area, for example.

The shopping area intrusion detection sensor 12 is configured to detect whether the shopping area is invaded. The shopping area intrusion detection sensor 12 may be, for example, an image sensor (set as the first image sensor), a photoelectric sensor in a reflective working mode, an ultrasonic ranging sensor, or a human body induction switch, etc. The corresponding detection information may be image information, ranging information , Spectral change information, etc. These sensors are described separately below.

The first image sensor may be an imaging device such as a camera. Adjust the installation position, angle, etc. of the first image sensor, and align the detection area (ie, shooting area) of the first image sensor with the shopping area to take an image of the shopping area. Based on the real-time image of the shopping area captured by the first image sensor, it can be determined whether the shopping area is invaded. For example, comparing the real-time image of the shopping area taken by the first image sensor with the reference image where the shopping area is not invaded, if a new object (such as a person) appears in the image of the shopping area, the shopping area is considered to be the object Invasion. In addition, other regional intrusion detection technologies can also be adopted to achieve intrusion detection in shopping areas.

The detection principle of the photoelectric sensor 30 in the reflective working mode is shown in FIG. 3. The photoelectric sensor 30 in the reflective operation mode includes a light emitter 31 and a light receiver 32. The light emitter 31 emits an optical signal, and the optical signal irradiated to a person is reflected and received by the light receiver 32. The distance between the person and the sales equipment is determined according to the time interval between the light signal received by the light receiver 32 and the light signal emitted by the light emitter 31, and whether the shopping area is invaded can be determined according to the distance.

The detection principle of the ultrasonic ranging sensor 40 is shown in FIG. 4. The ultrasonic ranging sensor 40 includes a transmitter 41 and a receiver 42. The transmitter 41 emits ultrasonic waves, and the ultrasonic waves that hit a person are reflected and received by the receiver 42. The distance between the person and the sales equipment is determined according to the time interval between the reception of ultrasonic waves by the receiver 42 and the transmission of ultrasonic waves by the transmitter 41, and whether the shopping area is invaded can be determined according to the distance.

The detection principle of the human body induction switch 50 is shown in FIG. 5. The human body induction switch 50 can be realized based on infrared technology, for example. When a person enters the sensing range, the human sensing switch 50 can detect changes in the infrared spectrum. Therefore, when the human body induction switch 50 detects the change of the infrared spectrum, it is considered that the shopping area is invaded.

The boundary area passing detection sensor 13 is configured to detect whether the boundary area between the placement area of the item to be sold and other areas has been crossed. The boundary area passing detection sensor 13 may be, for example, an image sensor (set as a second image sensor) or a photoelectric sensor in an on-beam operation mode, and the corresponding detection information may be image information, signal reception information, and the like. These sensors are described separately below.

The second image sensor may be an imaging device such as a camera. Adjust the installation position, angle, etc. of the second image sensor so that the detection area (ie, the shooting area) of the second image sensor is aligned with the boundary area, so as to take an image of the boundary area. Based on the image of the boundary area captured by the second image sensor, it can be determined whether the boundary area is crossed. For example, if images of at least two frames of the boundary area are continuously obtained, if an object (such as a person or a hand) appears on one side of the boundary area in some frames and on the other side of the boundary area in other frames, the boundary is considered The area is crossed by the object. At the same time, the direction in which the boundary area is crossed can also be determined. For example, in the time period T1, the object appears on one side of the boundary area a, and in the next later time period T2, the object appears on the other side b of the boundary area, then the boundary area is traversed from a to b direction.

The detection principle of the photoelectric sensor 60 in the beaming mode is shown in FIG. 6. The photoelectric sensor 60 in the contra-beam working mode includes several groups of light emitters 61 and light receivers 62 arranged in pairs. The light emitter 61 and the light receiver 62 provided in pairs form a light curtain covering the boundary area. When the light curtain is not blocked, the light signal emitted by the light emitter 61 can be received by the corresponding light receiver 62. When the light curtain is blocked, the light signal emitted by the light emitter 61 cannot be received by the corresponding light receiver 62. Therefore, according to the light receiver 62 receiving the optical signal, it can be determined whether the boundary area is passed through.

The shopping information detection sensor 14 is configured to detect the item information purchased by the user from the vending device where the item to be sold is placed. The shopping information detection sensor 14 may be, for example, a gravity sensor, an image sensor (set as the third image sensor), or a combination thereof.

The gravity sensor may be provided under each tray of the vending equipment, for example, to detect the gravity information of the items in the tray. According to the change of the gravity of the item, the item can be identified. For example, if the gravity of an item for sale detected by a gravity sensor decreases, and the amount of decrease in gravity is consistent with the gravity of an item a in each item of the tray (or each item of the sales equipment), then it is considered Item a is taken away, the user wants to purchase item a, and item a can be added to the user's virtual shopping cart for settlement. If the gravity of the item for sale detected by a gravity sensor increases, and the increase in gravity corresponds to the gravity of an item b among the items of the vending device, the item b is considered to be put back, and the user does not want to purchase the item b. The item b can be deleted from the user's virtual shopping cart.

For example, the third image sensor may be installed in a position in the vending device where the item can be picked and placed. The image captured by the third image sensor is identified to determine the items in the image. For example, the image features of each item for sale are obtained, and the image features of the items in the image to be recognized are compared with the image features of the items for sale. The higher the similarity, the items in the image to be recognized are the corresponding items for sale The greater the possibility. For example, the items in the image to be recognized are determined as the items with the highest similarity for sale. In some embodiments, for example, a double convolutional neural network may be used for item identification. The image to be detected is input into the first double convolutional neural network to detect whether there is an item in the image. If an item is detected, the item area is cropped and input into the second convolutional neural network to detect what kind of item. In addition, the trajectory of the item can be further tracked through the multi-frame image of the item, and according to the trajectory, it can be determined whether the item is the removed item that the user wants to purchase or the returned item that the user does not want to purchase.

In addition, the above item gravity recognition and item image recognition can also be combined. For example, compare whether the items identified by gravity and the items identified by the image are consistent. If they match, the confidence of the item identification result is relatively high. If the image recognition result includes multiple items with small confidence gaps, but the gravity of these items may be different, at this time, the amount of gravity change can be combined to determine the item that matches the gravity change in the image recognition result as the final recognition. result. If a certain gravity change corresponds to multiple items, but the images of these items may be different, at this time, the image recognition result can be combined to select the item matching the image recognition result in the gravity recognition result as the final recognition result.

Based on the detection information of at least one of the above-mentioned sensors 12, 13, and 14, the processor 11 may determine the occurrence of a predetermined event, and may further trigger operations associated with the predetermined event.

In some embodiments, the processor 11 is configured to determine the occurrence of an event that the user enters or leaves the shopping area based on the detection information of the shopping area intrusion detection sensor 12.

For example, based on the detection information of various shopping area intrusion detection sensors 12, it can be determined whether the shopping area is invaded. If the shopping area is invaded by a user at a certain time t1, it is considered that the user enters the shopping area at time t1. At a certain time t2, when the shopping area is invaded by the user, the event that the user leaves the shopping area occurs at time t2.

In some embodiments, the processor 11 is configured to determine the occurrence of an event in which the boundary area is traversed or an event in which the boundary area is traversed based on the detection information of the boundary area crossing detection sensor 13.

As described above, based on the detection information of the sensor 13 of the second image sensor type, the occurrence of the event in which the boundary area is traversed and the event in which the boundary area is traversed can be determined. Based on the detection information of the sensor 13 of the photoelectric sensor 60 type of the collimated working mode, the occurrence of the event that the boundary area is traversed can be determined.

In some embodiments, according to business requirements, the association relationship between the predetermined event and the operation may be set as:

1) The event that the user enters the shopping area is related to the item identification operation, that is, the item identification operation is performed after the user enters the shopping area. Therefore, the calculation amount of item identification is reduced, and the item identification speed is improved.

As mentioned above, for example, item identification is based on at least one of item gravity and item image, that is, item identification based on item gravity, item identification based on item image, and item identification based on the combination of item gravity and item image . Here and in the following, the specific item identification method will not be described in detail.

2) The event that the user enters the shopping area is associated with caching the sensing information used for item identification, that is, after the user enters the shopping area, the sensory information such as item image information or item gravity information used for item identification is cached . Therefore, it can not only meet the needs of item identification, but also reduce the amount of cached information.

3) The event that the user leaves the shopping area is related to the item settlement operation, that is, the item settlement operation is performed after the event that the user leaves the shopping area.

Thus, the settlement operation in the sales business scenario is smoothly carried out. Some related technologies use the closing signal of the door of the vending equipment to trigger the settlement operation, and an abnormal situation that cannot be settled will occur. For example, the user did not close the door of the vending equipment after purchasing the product, or the door lock could not be closed normally due to some mechanical or electrical failure.

4) The event that the boundary area is crossed is related to the operation of identifying the item or the operation of adding or deleting the purchased item, that is, the event of the boundary area is crossed, and then the operation of identifying the item or the addition or deletion of the purchased item is performed .

As mentioned earlier, the boundary area is an area very close to the item to be sold, and the image of the item being taken away or put back densely appears near the event where the boundary area is traversed. Therefore, the calculation amount of item identification is reduced, and the item identification speed is improved.

5) The operation in which the boundary area is recognized by the event of the crossing direction is associated with the item or the operation of adding or deleting the associated purchased item. That is, after the event that the boundary area is crossed in the direction occurs, the operation of item identification or the addition or deletion of purchased items is performed. Therefore, the calculation amount of item identification is reduced, and the item identification speed is improved.

In addition, based on the occurrence time of the event that the boundary area is traversed or based on the occurrence time of the event that the boundary area is traversed, the item images within the preset range are retrieved for item identification or addition of purchased items Or delete. For example, the image of the item within the preset time range before and after the relevant event occurs, or the image of the item within the preset frame number before and after the relevant event occurs. Thus, the calculation amount of item identification is further reduced.

In addition, the processor 11 is configured to determine the occurrence of an abnormal event based on the detection information of at least one of the shopping area intrusion detection sensor 12 and the boundary area passing detection sensor 13 and the shopping information detection sensor 14, and trigger the association of the abnormal event Operation.

Abnormal events and their associated operations can be set according to business needs. There are many cases of abnormal events and their associated operations. Here are some examples. When identifying purchased items based on the shopping information detection sensor 14, if the item identification result based on the item gravity is inconsistent with the item identification result based on the item image, it indicates that an item identification abnormal event has occurred, which can be reported to the staff or the purchase user. Manually select purchased items. Alternatively, based on the detection information of the boundary area passing through the detection sensor 13 to determine that the boundary area has not been crossed, item identification may not be performed, for example, item identification based on the image of the item may not be performed, thereby effectively shielding abnormalities such as image misdetection or image shake Misidentification caused by the event. Based on the detection information of the shopping area intrusion detection sensor 12, it is recognized that dangerous goods have entered the shopping area, and an alarm can be given. Based on the detection information of the boundary area passing through the detection sensor 13 when it is recognized that a dangerous item is put back into the selling equipment, an alarm can be given.

7 shows a schematic flowchart of some embodiments of a sales method applied to a sales device of the present disclosure.

As shown in FIG. 7, the method 70 of this embodiment includes: steps 71-74. This method may be executed by the processor 11, for example.

Step 71: Obtain the detection information of whether the shopping area is invaded.

For example, the processor 11 receives and analyzes the detection information of whether the shopping area is invaded by the shopping area intrusion detection sensor 12, and accordingly, it can be known whether the shopping area is invaded.

Step 72: Obtain the detection information of whether the boundary area between the area where the item to be sold is placed and other areas is passed through.

For example, the processor 11 receives and analyzes the detection information of whether the boundary area passed by the boundary area crossing detection sensor 13 has been crossed, based on which information such as whether the boundary area has been crossed and the direction of crossing can be known.

Step 73: Determine the occurrence of a predetermined event based on at least one piece of detection information.

As described above, the processor 11 determines the occurrence of the event that the user enters or leaves the shopping area based on the detection information of whether the shopping area is invaded, and based on the detection information of whether the boundary area between the area where the item to be sold is placed and other areas is crossed To determine the occurrence of the event where the boundary area is traversed or the direction where the boundary area is traversed.

Step 74: Trigger the operation associated with the predetermined event to implement automatic vending.

As described above, the processor 11 may, for example, perform the operation associated with the event according to the preset association relationship between the predetermined event and the operation. These operations are, for example, operations related to sales, such as item identification, addition or deletion of purchased items, and settlement.

For example, when an event that a user enters a shopping area occurs, an operation of buffering sensing information such as images or gravity used for item identification is triggered. In the event that the user leaves the shopping area, an item settlement operation is triggered. In the event of an event that the boundary area is traversed, the operation of item identification is triggered. In the event of an event that the boundary area is crossed in the direction, trigger the operation of item identification, and the operation of adding or deleting purchased items. If the boundary area is crossed away from the vending equipment, it means that the user may have taken the item from the sales device and can add the purchased items. If the boundary area is crossed through the direction toward the vending equipment , Indicating that the user may return the item to the vending device, and can perform the delete operation of the purchased item.

In addition, in the case where the type of the sensor can only determine that the boundary area is traversed, but cannot determine the direction in which the boundary area is traversed, the operation of taking the item or taking it back can be determined by combining at least one of the item gravity and the item image , Based on the determined operation of taking the item or putting it back, triggering the addition or deletion of the purchased item.

For example, if the boundary area is crossed, the gravity of the items for sale detected by the gravity sensor decreases, indicating that the user may take the items from the sales device, trigger the addition operation of the purchased items, and add the removed items to the user's virtual shopping In the car. The boundary area is crossed, and the gravity of the items for sale detected by the gravity sensor increases, indicating that the user may return the items to the sales device, trigger the deletion operation of the purchased items, and delete the returned items from the user's virtual shopping cart.

For example, the demarcation area is crossed, and the direction of the item trajectory based on the image recognition is away from the vending device, indicating that the user may take the item from the vending device, and can perform the addition operation of the purchased item, if based on the image recognition The direction of the item's trajectory is the direction toward the vending device, indicating that the user may put the item back into the vending device and can perform the deletion operation of the purchased item.

In addition, based on the occurrence time of the event where the boundary area is traversed or the occurrence time of the event where the boundary area is traversed, the item image within the preset range can be retrieved, and the item recognition can be performed based on the retrieved item image, or , Based on the retrieved item image combined with item gravity for item identification. Thus, the calculation amount of item identification is further reduced.

FIG. 8 shows a schematic flowchart of some embodiments of a sales method applied to a sales device of the present disclosure.

As shown in FIG. 8, the method 80 of this embodiment includes: steps 81-89.

Step 81: Use the detection information of the shopping area intrusion detection sensor 12 to determine whether the user enters the preset shopping area. If no user enters the preset shopping area, continue to perform this judgment step. If the user enters the preset shopping area, then Step 82 can be performed.

Step 82, start to cache the detection information of the shopping information detection sensor 14.

When the shopping information detection sensor 14 is a gravity sensor, the item gravity detection information is cached, and when the shopping information detection sensor 14 is the third image sensor, the item image detection information is cached.

Since item recognition based on item gravity consumes less computing resources and item identification based on item images consumes more computing resources, therefore, step 84a can be directly performed after step 82, and of course, steps can be continued after step 82 83. In the case where the boundary area is crossed by human hands, step 84a is executed again. The latter can save more computing resources than the former.

Step 83: Use the detection information of the boundary area crossing detection sensor 13 to determine whether the boundary area is crossed by human hands and the direction of crossing. If the boundary area is crossed by human hands, step 84b is executed. If the boundary area is not crossed by human hands, Go to step 85.

Step 84a: Perform item identification based on the cached item gravity detection information. The article identification method based on the gravity of the article refers to the foregoing, and will not be repeated here.

Step 84b, the boundary area is crossed by hand, indicating that the user is about to operate the item to be sold, for example, taking the item to be sold, putting it back into the item to be sold, etc. At this time, the image of the item within the preset time range of the cache can be retrieved, Article recognition is performed based on the retrieved article images.

For example, if the direction of the human hand crossing is close to the sales equipment, it may indicate that an item may be placed, and the image of the first B1 second of the human hand crossing the boundary area and the A1 second of the rear crossing the boundary area (B1>A1) is carried out. Item identification. If the direction of the human hand crossing is away from the sales equipment, it means that the item may be taken, and the images of the first B2 seconds of the human hand crossing the boundary area and the A2 seconds of the rear crossing the boundary area (B2<A2) are retrieved for item recognition .

Since it only needs to process items within a certain period of time, it can increase the speed of item identification and save computing resources.

In step 85, the various recognition results in steps 83, 84a and 84b are fused.

An exemplary fusion method is that if the boundary area is not penetrated, item identification is not performed, for example, item identification based on item images is not performed, thereby effectively shielding misidentification caused by abnormal events such as image misdetection or image shake .

Yet another exemplary fusion method is to combine the item recognition result based on the gravity of the item and the item recognition result based on the image of the item to comprehensively determine the item recognition result. For the comprehensive determination method, refer to the foregoing, and will not be repeated here.

If the various recognition results are consistent, for example, the boundary area is crossed and the gravity recognition method and the image recognition method are consistent, then the removed items are added to the virtual shopping cart, or the returned items are purchased from the virtual shopping Cart deleted.

If the various recognition results are inconsistent, for example, the article recognition results of the gravity recognition method and the image recognition method are inconsistent, mark them as abnormal events, and save the relevant image or gravity detection information and identification information as risk data for subsequent risk control .

The item recognition method based on fusion of multiple sensors has stronger robustness and accuracy.

Step 86: Use the detection information of the shopping area intrusion detection sensor 12 to determine whether the user leaves the preset shopping area. If you have not left the preset shopping area, continue with the steps related to item identification. If you leave the preset shopping area, go to step 87.

In step 87, it is determined whether there is an abnormal event, for example, an identification abnormal event in which various recognition results are inconsistent, or other abnormal events.

Step 88, if there is an abnormal event, the corresponding processing can be performed according to the operation corresponding to the abnormal event. Exemplary abnormal event processing methods refer to the foregoing, and will not be repeated here.

Step 89: If there is no abnormal event, the items in the virtual shopping cart can be settled to complete the sales process.

Based on the detection information of whether the shopping area has been invaded, and the detection information of whether the boundary area between the item placement area and other areas has been crossed, to determine the occurrence of shopping related events such as the user entering or leaving the shopping area, the boundary area being crossed, Combined with the detected shopping information, the corresponding shopping-related operations are triggered to realize automatic vending. As a result, at least one problem such as abnormal settlement in the automatic vending business, large amount of calculation of item identification, and the like is improved.

Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code .

The above are only preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the protection of the present disclosure Within range.

Claims

A vending device, including:

processor,

as well as

At least one of the shopping area intrusion detection sensor and the boundary area passing through the detection sensor,

Wherein, the processor is electrically connected to the shopping area intrusion detection sensor, and the processor is electrically connected to the boundary area through the detection sensor;

The shopping area intrusion detection sensor is configured to detect whether the shopping area is invaded;

The boundary area passing through the detection sensor is configured to detect whether the boundary area between the placement area of the item to be sold and other areas is crossed;

The processor is configured to determine the occurrence of a predetermined event based on the detection information of at least one of the shopping area intrusion detection sensor and the boundary area penetration detection sensor, and trigger an operation associated with the predetermined event.
The vending device according to claim 1, wherein:

The processor is configured to determine the occurrence of an event that the user enters or leaves the shopping area based on the detection information of the shopping area intrusion detection sensor;

and / or,

The processor is configured to determine the occurrence of an event in which the boundary area is traversed or an event in which the boundary area is traversed based on the detection information of the boundary area through the detection sensor.
The vending device according to claim 1, wherein the association relationship between the predetermined event and the operation is configured as at least one of the following relationships:

The event that the user enters the shopping area is related to the operation of item identification or the operation of associating the cached sensing information used for item identification;

The event that the user leaves the shopping area is related to the operation of item settlement;

The event that the boundary area is crossed is related to the operation of identifying the item or the operation of adding or deleting the related purchased item;

The demarcation area is associated with the operation of item recognition in the event of crossing direction or the operation of adding or deleting the purchased item.
The vending device according to claim 3, wherein

The event that the user enters the shopping area is associated with an item recognition operation based on at least one of item gravity and item image;

Or, the event that the demarcation area is traversed is associated with the operation of item recognition based on at least one of item gravity and item image, or with the addition or deletion of purchased items implemented with combination of item gravity and at least one item image operating;

Or, the event that the demarcation area is crossed in the direction is associated with the operation of item recognition based on at least one of the item gravity and the item image, or the addition of the purchased item or the combination of the item gravity and the item image. Delete operation.
The vending device according to claim 3, wherein

The processor is configured to retrieve an image of an item within a preset range for item identification or purchase based on the time of occurrence of the event where the boundary area is traversed or the event of the direction where the boundary area is traversed Add or delete items.
The vending device according to claim 1, wherein:

The processor is configured to determine the occurrence of an abnormal event based on the detection information of at least one of the shopping area intrusion detection sensor and the boundary area passing detection sensor and the shopping information detection sensor, and trigger the association of the abnormal event Operation,

Wherein, the shopping information detection sensor is configured to detect the item information purchased by the user from the vending device where the item to be sold is placed.
The vending device according to claim 1, wherein:

The boundary area is set according to a plane with an angle smaller than a preset angle with the front surface of the sales equipment where the items to be sold are placed, and the distance between the boundary area and the front surface of the sales equipment is smaller than the preset distance;

and / or,

The shopping area is set according to a preset area area on the location of the vending equipment.
The vending device according to claim 1, wherein:

The shopping area intrusion detection sensor is a first image sensor, a photoelectric sensor in reflective working mode, an ultrasonic ranging sensor, or a human body induction switch, wherein the detection area of the first image sensor is aligned with the shopping area;

and / or,

The boundary area passing through the detection sensor is a second image sensor, or a photoelectric sensor in a contra-beam operating mode, wherein the detection area of the second image sensor is aligned with the boundary area.
A sales method applied to a sales device, which is characterized by including:

Obtain the detection information of whether the shopping area is invaded;

Obtain the detection information of whether the boundary area between the area where the item to be sold is placed and other areas is crossed;

Determine the occurrence of a predetermined event based on at least one detection information; and

The operation associated with the predetermined event is triggered in order to realize sales.
The method of claim 9, wherein:

Based on the detection information of whether the shopping area is invaded, it is determined that the user enters or leaves the shopping area.
The method of claim 10, wherein

In the case of an event where a user enters a shopping area, trigger the operation of caching the sensing information used for item identification;

In the event that the user leaves the shopping area, an item settlement operation is triggered.
The method according to any one of claims 9-11, wherein

Based on the detection information of whether the boundary area between the placement area of the item to be sold and the other area is crossed, the occurrence of the event that the boundary area is crossed or the direction in which the boundary area is crossed is determined.
The method of claim 12, wherein

In the event of an event where the boundary area is crossed, trigger the item identification operation;

In the event that an event occurs in the direction in which the boundary area is crossed, trigger the operation of item identification, and the operation of adding or deleting purchased items.
The method of claim 13, wherein:

Combining at least one of the item gravity and the item image to determine the operation of taking the item or returning the item, based on the determined operation of taking the item or returning the item, triggering the addition or deletion of the purchased item.
The method of claim 13, wherein:

Based on the occurrence time of the event that the boundary area is traversed or the occurrence time of the event of the direction that the boundary area is traversed, an item image within a preset range is retrieved, and item recognition is performed based on the retrieved item image or in combination with the gravity of the item.
A non-transitory computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the sales method of any one of claims 9-15.