WO2021098373A1

WO2021098373A1 - Device, method and apparatus for adjusting and controlling shelf

Info

Publication number: WO2021098373A1
Application number: PCT/CN2020/117447
Authority: WO
Inventors: 周剑; 明泉水; 廖耿耿; 宋启恒; 郑辉
Original assignee: 支付宝(杭州)信息技术有限公司
Priority date: 2019-11-18
Filing date: 2020-09-24
Publication date: 2021-05-27
Also published as: CN112990138A; CN112990138B; CN110991273B; CN110991273A

Abstract

A device, method and apparatus for adjusting and controlling a shelf, which are applied to a vending cabinet, with a storage space inside the vending cabinet being divided into a plurality of storage layers by means of shelves. The device for adjusting and controlling a shelf comprises: cameras (11) that are arranged on the top side of all storage layers, a main control module (12) and a lifting module (13), wherein when a cabinet door of a vending cabinet is opened, each camera (11) collects a first image of the storage layer where the camera itself is located, and when the cabinet door of the vending cabinet is closed, the camera collects a second image and a third image of the storage layer where the camera itself is located, the first image and the second image comprise planar image information of the storage layer, and the third image comprises depth image information of the storage layer; the main control module (12) determines a shelf to be moved and movement information of said shelf according to the images collected by the camera (11); and the lifting module (13) is connected to all shelves, and controls the movement of said shelf according to the movement information of said shelf.

Description

Laminate control equipment, method and device

Technical field

This application relates to the technical field of the Internet of Things, and in particular to a laminate control equipment, method and device.

Background technique

With the rapid development of the Internet of Things technology, self-service vending containers that do not require human guards and can sell goods to consumers around the clock are widely used. They process the images collected by the cameras installed inside to identify the goods purchased by consumers. And then settle the fee.

Currently, self-service vending containers are usually equipped with multiple storage layers, and the space of each storage layer is fixed. However, the storage layer with a fixed space size not only has a low space utilization rate, but also may cause consumers to purchase goods. When the original product placement is disrupted, the tall and short products block each other, resulting in blind spots that cannot be captured by the camera, which leads to problems such as inaccurate identification of the products purchased by consumers.

Summary of the invention

The purpose of one or more embodiments of this specification is to provide a control device, method, and device that can flexibly control the position of the shelf in the sales container based on the image collected by the camera, which can not only improve the utilization of storage space, but also And it can increase the distance between the camera in the storage layer where the product is operated and the product, thereby reducing the blind spot and corner of the camera, improving the accuracy of the image collected by the camera, and improving the identification of the purchased product Accuracy of recognition results.

In order to solve the above technical problems, one or more embodiments of this specification are implemented as follows.

One or more embodiments of this specification provide a laminate control device, which is applied to a sales container, wherein the storage space inside the sales container is divided into multiple storage layers by the laminate, and the device includes: A camera on the top side of each of the storage layers; the camera collects the first image of the storage layer where it is when the door of the sales container is opened, and when the door of the sales container is closed Acquire a second image and a third image of the storage layer where oneself is located; wherein the first image and the second image include planar image information of the storage layer, and the third image includes the storage layer The main control module, which determines the first target storage layer where the operated commodity is located according to the first image and the second image collected by each of the cameras; and, according to each of the cameras The acquired third image and the first target storage layer determine the to-be-moved floor in the floor and the moving information of the to-be-moved floor; the lifting module, which is connected to each of the floor Connected, and control the movement of the layer to be moved according to the movement information of the layer to be moved.

One or more embodiments of the present specification provide a layer control method, which is applied to a sales container, wherein the storage space inside the sales container is divided into a plurality of storage layers by the layer, and the method includes: obtaining The first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first image, The second image and the third image are both obtained by a camera arranged on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the second image Including the plane image information of the storage layer, the third image includes the depth image information of the storage layer; according to the first image and the second image, determine the first target storage where the operated commodity is located Object layer; according to the third image and the first target storage layer, determine the layer to be moved in the layer and the movement information of the layer to be moved; according to the movement of the layer to be moved Information to control the movement of the layer to be moved.

One or more embodiments of this specification provide a layer control device, which is applied to a sales container, wherein the storage space inside the sales container is divided into multiple storage layers by the layer, and the device includes: Module, which acquires the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the The first image, the second image, and the third image are all obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the The second image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer; a first determination module, which, according to the first image and the second image, Determine the first target storage layer where the operated commodity is located; a second determination module, which determines the layer to be moved in the layer and the layer to be moved according to the third image and the first target storage layer The movement information of the moving floor; a control module, which controls the movement of the floor to be moved according to the movement information of the floor to be moved.

One or more embodiments of this specification provide a layer control device, including: a processor; and, a memory arranged to store computer-executable instructions that, when executed, cause the processor to : Acquire the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first The image, the second image, and the third image are all obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the first image The second image includes the plane image information of the storage layer, the third image includes the depth image information of the storage layer; according to the first image and the second image, the first image where the operated commodity is located is determined Target storage layer; according to the third image and the first target storage layer, determine the layer to be moved in the layer and the movement information of the layer to be moved; according to the layer to be moved To control the movement of the layer to be moved.

One or more embodiments of this specification provide a storage medium for storing computer-executable instructions that, when executed, realize the following process: Obtain each storage medium when the door of the sales container is opened. The first image of the object layer, and the second image and the third image of each storage layer when the cabinet door is closed; wherein, the first image, the second image, and the third image are all It is obtained by a camera set on the top side of each storage layer to collect the commodities placed in the storage layer where it is located; the first image and the second image include the plane image information of the storage layer, so The third image includes depth image information of the storage layer; according to the first image and the second image, the first target storage layer where the operated commodity is located is determined; according to the third image and the The first target storage layer determines the floor to be moved in the floor and the movement information of the floor to be moved; according to the movement information of the floor to be moved, the movement of the floor to be moved is controlled.

An embodiment of this specification realizes the flexible adjustment of the position of the shelf in the sales container, which not only improves the utilization rate of the storage space, but also increases the distance between the camera and the commodity in the storage layer where the commodity is operated, and reduces the distance between the camera and the commodity in the storage layer where the commodity is operated. The blind spots and blind spots of the camera can improve the accuracy of the images collected by the camera and the accuracy of the recognition results for identifying the purchased goods.

Description of the drawings

In order to more clearly describe the technical solutions in one or more embodiments of this specification, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only in this specification. For some of the described embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application scenario for self-service vending of vending containers provided by one or more embodiments of this specification;

2 is a schematic diagram of the module composition of a laminate control device provided by one or more embodiments of this specification;

Figure 3 is a schematic structural diagram of a lifting module provided by one or more embodiments of this specification;

FIG. 4 is a schematic diagram of the first flow of a layer control method provided by one or more embodiments of this specification;

FIG. 5 is a detailed diagram of step S104 provided by one or more embodiments of this specification;

6 is a schematic diagram of a matrix of binarized images provided by one or more embodiments of this specification;

FIG. 7 is a detailed diagram of step S106 provided by one or more embodiments of this specification;

FIG. 8 is a first detailed diagram of step S106-8 provided in one or more embodiments of this specification;

FIG. 9 is a second detailed diagram of step S106-8 provided in one or more embodiments of this specification;

FIG. 10 is a schematic diagram of the second flow of a layer control method provided by one or more embodiments of this specification;

11 is a schematic diagram of the module composition of a laminate control device provided by one or more embodiments of this specification;

Figure 12 is a schematic structural diagram of a laminate control device provided by one or more embodiments of this specification;

Wherein, the reference signs are explained as follows:

10: Shelves;

13-1: Drive unit;

13-2: Guide rail;

13-3: scroll wheel;

13-4: Output terminal.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to compare the technical solutions in one or more embodiments of this specification. The technical solution is described clearly and completely. Obviously, the described embodiments are only a part of the embodiments in this specification, rather than all the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of this application.

One or more embodiments of this specification provide a layer control device, method, and device, which can be applied to a sales container, and can flexibly control the position of each layer in the sales container based on the image collected by the camera. Improve the utilization of storage space, and increase the distance between the camera in the storage layer where the product is operated and the product, thereby reducing the blind spots and blind spots of the camera, and improving the accuracy of the image collected by the camera , And improve the accuracy of the recognition results that identify the purchased goods.

Figure 1 is a schematic diagram of an application scenario for self-service vending of vending containers provided by one or more embodiments of this specification. As shown in Fig. 1 (a), for a vending container with a fixed floor height, the consumer scans Before using the QR code to open the cabinet door to purchase goods, the goods in each storage layer of the selling container are placed in an orderly manner; as consumers will pay attention to the shelf life, composition and other information of the goods when buying goods, they usually do Take multiple products for viewing and comparison, but consumers often do not pay special attention to the placement of the products, and randomly put the unwanted products back to the free position of the storage layer, and finally sell them after the consumers buy the products. The goods in the container will be in the state shown in Figure 1(b). The goods are placed in a disorderly manner, and the tall and short products are blocked from each other. Since the height of each storage layer in the sales container is fixed, it is in a limited space. Here, the disordered and occluded state of the product will cause a blind spot and a blind spot for the camera, which will lead to a decrease in the accuracy of the recognition result when the product purchased by the consumer is recognized based on the image collected by the camera. Based on this, one or more embodiments of this specification provide a layer control device, method and device, which can flexibly adjust the position of each layer based on the image collected by the camera and the space size of each storage layer in the sales container. For ease of description, in the order from top to bottom, each storage layer is marked as storage layer 1, storage layer 2, storage layer 3, etc. After layer adjustment is performed on the basis of Figure 1 (b) The schematic diagram of is shown in Figure 1(c), it can be seen that by adjusting the layer plate, the space of the storage layer 2 can be enlarged, and the distance between the camera in the storage layer 2 and the goods in the storage layer 2 is increased , The shooting angle of the camera is increased, and the blind spot of the camera is reduced, so that the accuracy of the recognition result is improved when the product purchased by the consumer is recognized according to the image captured by the camera. Further, in order to determine the movement information of the laminate, in one or more embodiments of this specification, the camera is a 3D camera, and each video frame in the video stream captured by it includes an image containing planar image information and an image containing depth image information. Image; Among them, depth image information such as three-dimensional space position, size and other information. The following is a detailed description of the laminate control equipment, method, and device provided in one or more embodiments of this specification.

Figure 2 is a schematic diagram of the module composition of a laminate control device provided by one or more embodiments of this specification, which is applied to a sales container, where the storage space inside the sales container is divided into multiple storage layers by the laminate, such as As shown in Figure 2, the device includes the following.

A camera 11 arranged on the top side of each storage layer; the camera 11 collects the first image of the storage layer where it is when the door of the sales container is opened, and collects the storage where it is when the door of the sales container is closed The second image and the third image of the layer; wherein the first image and the second image include the plane image information of the storage layer, and the third image includes the depth image information of the storage layer.

The main control module 12, which determines the first target storage layer where the operated commodity is located according to the first image and the second image collected by each camera 11; and, according to the third image collected by each camera 11 and the determined second image For a target storage layer, determine the layer to be moved in the layer and the movement information of the layer to be moved.

The lifting module 13 is connected with each layer, and controls the movement of the layer to be moved according to the movement information of the layer to be moved.

The shelf control device provided by one or more embodiments of this specification can flexibly control the position of the shelf in the sales container based on the image collected by the camera, which can not only improve the utilization of storage space, but also increase the number of operations being performed. The distance between the camera in the storage layer of the product and the product, thereby reducing the blind spots and blind spots of the camera, improving the accuracy of the image collected by the camera, and improving the accuracy of the recognition result of identifying the purchased product.

Since the operated product is recognized based on the images captured by the camera before and after the consumer purchases the product, that is, the recognition is based on the image captured by the camera before the door of the selling container is opened and after the door is closed; Before and after the product is purchased, the status of the product has changed due to some malicious operations (such as shaking the selling container, etc.) or other factors, so that the problem of the operated product cannot be correctly identified. In one or more embodiments of this manual: Main The control module 12 sends a first image acquisition request to each camera 11 according to the first detection time of the door opening signal when detecting the door opening signal of the selling container door being opened; and, when detecting that the door of the selling container is When the door-closing signal is closed, the second image acquisition request is sent to each camera 11 according to the second detection time of the door-closing signal; correspondingly, the camera 11 receives the first image acquisition request according to the first detection time. Obtain the corresponding video frame from the video stream, and extract the first image from the obtained video frame; and, when receiving the second image acquisition request, obtain the corresponding video frame from the captured video stream according to the second detection time, from The acquired video frame extracts the second image and the third image.

Optionally, the camera 11, upon receiving the first image acquisition request, acquires the previous video frame of the video frame corresponding to the first detection time from the captured video stream, and extracts the first image from the acquired video frame; or , The camera 11, when receiving the first image acquisition request, uses the first detection time as the expiration time of the first preset duration, determines the start time corresponding to the first preset duration, and displays it in the captured video stream Obtain the video frame corresponding to the start time in, and extract the first image from the acquired video frame; or, when the camera 11 receives the first image acquisition request, the first detection time is set as the expiration time of the first preset duration , Acquire any video frame within the first preset duration in the captured video stream, and extract the first image from the acquired video frame; and, the camera 11, upon receiving the second image acquisition request, Obtain the next video frame of the video frame corresponding to the second detection time from the video stream, and extract the second image and the third image from the acquired video frame; or, when the camera 11 receives the second image acquisition request, it uses the first 2. The detection time is the start time of the second preset duration, the deadline corresponding to the second preset duration is determined, and the video frame corresponding to the deadline is obtained from the captured video stream, and from the acquired video frame Extract the second image and the third image; or, when the camera 11 receives the second image acquisition request, the second detection time is used as the start time of the second preset duration, and the second image is acquired from the captured video stream. Any one of the video frames within the preset duration, extract the second image and the third image from the acquired video frames; or, when the camera 11 receives the second image acquisition request, the second detection time is the second preset duration At the start time, any two video frames within the second preset duration are acquired from the captured video stream, the second image is extracted from one of the acquired video frames, and the third image is extracted from the other acquired video frame.

Among them, the first preset duration and the second preset duration are both very short. The first preset duration and the second preset duration may be the same or different. They can be set according to actual needs; as an example, The first preset duration and the second preset duration are the same, and both are 1 second. It should be pointed out that when the camera 11 acquires the first image, it can also acquire the fifth image containing the depth image information of the storage layer at the same time. Thus, video frames with a short interval from the first detection time are acquired according to the first detection time to extract the first image, and video frames with a short interval from the second detection time are acquired according to the second detection time to extract the second image and The third image, and it is difficult for consumers or other users to perform malicious operations within this short time interval. Therefore, the accuracy of the first image, the second image, and the third image can be greatly ensured, so as to make subsequent determinations The first target storage layer where the operated product is located, the determination of the moving information of the floor to be moved, and the determination of the purchased product provide an effective data basis.

To ensure the effective transmission of images, a transmission channel is established between each camera 11 and the main control module 12. When the camera 11 obtains the first image, it sends the first image to the main control module 12 through the corresponding transmission channel, and When the camera 11 acquires the second image and the third image, it sends the second image and the third image to the main control module 12 through the corresponding transmission channel; the main control module 12 receives the second image, based on the first image And the second image determines the first target storage layer where the operated commodity is located.

Specifically, in one or more embodiments of this specification, the main control module 12 determines the first image and the second image corresponding to each storage layer; and, the first image and the second image corresponding to each storage layer The two images are subjected to differential processing to obtain the differential image corresponding to each storage layer; the differential image corresponding to each storage layer is binarized to obtain the binarized image corresponding to each storage layer; The target binarized image that meets the preset conditions is determined in the image; the storage layer corresponding to the target binarized image is taken as the first target storage layer where the operated commodity is located.

By performing differential processing on the first image and the second image corresponding to each storage layer to weaken the similar parts of the first image and the second image corresponding to each storage layer, the difference between the first image and the second image can be obtained. The difference image of the difference between the two values; the difference image is binarized to obtain a binarized image with only black and white visual effects, and the interference of other noises is removed, so that the binarized image can be accurately determined The first target storage layer where the operated commodity is located. Further, determine the first image and the second image corresponding to each storage layer, perform differential processing on the first image and the second image, perform binarization processing on the differential image, and determine that the binarized image is consistent with the prediction The specific process of setting the conditional target binarization image will be described in detail later, please refer to the related description later, and the description will not be repeated here.

In order to expand the space of the first target storage layer and expand the distance between the camera in the first target storage layer and the goods, so that the camera in the first target storage layer can collect clear and complete images, first Determine the floor to be moved and the moving information of the floor to be moved;

Specifically, in one or more embodiments of this specification, the main control module 12 determines the third image corresponding to each storage layer; and, among the commodities included in the third image corresponding to each storage layer, Obtain the target product with the smallest vertical distance between the top of the product and the top of the storage layer where the product is located; use the vertical distance between the top of the target product and the top of the storage layer where the target product is located as the remainder of the storage layer where the target product is located Floor height: According to the remaining floor height of the storage layer, determine the floor to be moved in the floor and the movement information of the floor to be moved.

In practical applications, according to the remaining layer height of the storage layer, all the layers with moving space can be determined and used as the layer to be moved, so as to maximize the space of the first target storage layer, so that the first target storage layer can be maximized. The distance between the camera in the object layer and the product is maximized, so that the camera can collect clear and complete images; according to the remaining height of the storage layer, a part of the floor with moving space can be determined and used as the floor to be moved , On the basis that the distance between the camera in the first target storage layer and the product reaches the preset distance that can collect clear and complete images, and do not move too many layers to ensure safety and stability.

Specifically, in one or more embodiments of this specification, if the main control module 12 determines that there is a non-zero storage layer based on the remaining layer height and the preset safety distance of each storage layer except the first target storage layer For the storage layer with a movable distance, the floor to be moved is determined according to the positional relationship between the storage layer with a non-zero movable distance and the first target storage layer; and, according to the position of the storage layer to be moved and the first target storage layer The position relationship of the layers to determine the moving direction of the layer to be moved; according to the movable distance of the storage layer corresponding to the layer to be moved, the actual moving distance of the layer to be moved is determined; the moving direction of the layer to be moved is compared with the actual The movement distance is used as the movement information of the floor to be moved; or, if the main control module 12 determines that there is a distance according to the remaining height of each storage layer except the first target storage layer and the preset safety distance A second target storage layer with the closest target storage layer and the total movable distance not less than the preset distance is determined according to the second target storage layer; and, according to the second target storage layer and the second target storage layer; The position relationship of a target storage layer determines the moving direction of the layer to be moved; according to the movable distance of the second target storage layer, the actual moving distance of the layer to be moved is determined; the moving direction of the layer to be moved is compared with the actual The moving distance is used as the moving information of the layer to be moved.

By determining the movement information of the floor to be moved, the subsequent accurate control of the floor to be moved is performed based on the movement information, without damaging part of the products due to the different heights of the products in the same storage layer.

After the movement information of the floor to be moved is determined, the movement of the floor to be moved can be controlled according to the movement information; specifically, in one or more embodiments of this specification, the lifting module 13 includes: a driving unit 13-1, a guide rail 13-2, and roller 13-3; among them, the roller 13-3 is arranged on the side of each layer, corresponding: the main control module 12, according to the moving information of the layer to be moved and the label of the layer, send the moving instruction of the layer To the drive unit 13-1; the drive unit 13-1, its output terminal 13-4 is connected to each layer, when receiving a layer movement instruction, according to the movement information to drive the corresponding layer plate identification to be moved through it The roller 13-3 provided on the side moves along the guide rail 13-2.

It can be seen that by setting the lifting module 13, the adjustment of the position of the middle deck of the selling container is realized.

In order to ensure that the layer to be moved does not slide down due to gravity after it is moved, in one or more embodiments of this specification, the driving unit 13-1 is a structure with a limit function, such as a pneumatic cylinder, a hydraulic cylinder, and a ball wire. Structures such as bars, which can accurately control the actual moving distance of the layer to be moved in the corresponding direction according to the movement information, and then remain in the corresponding position without sliding down the guide rail 13-2 through the roller 13-3 due to gravity. . In practical applications, the drive unit 13-1 can also be a structure without a limit function. If it is an ordinary motor, the lifting module 13 also includes a limit unit 13-5, which can be set at The side surface of the laminate can also be arranged on the guide rail 13-2. After the driving unit 13-1 controls the laminate to be moved to move the actual moving distance in the corresponding direction, the limit unit 13-5 is controlled to limit the position.

Further, the roller 13-3 can be arranged on the left side, and/or the right side, and/or the rear side of the shelf. Correspondingly, the guide rail 13-2 can be arranged on the left side of the storage space of the sales container, and /Or right side, and/or back side.

Furthermore, the guide rails 13-2 can be set in one-to-one correspondence with the laminates, and the same guide rails 13-2 can also be shared by all the laminates. In actual applications, it can be set according to the needs; it needs to be pointed out that When the guide rails 13-2 and the laminates are arranged in one-to-one correspondence, the positions of the rollers 13-3 on the sides of the different laminates can be the same or different. For ease of description, the layers are arranged in order from top to bottom. Marked as layer 1, layer 2, layer 3, etc., for example, the roller 13-3 arranged on the side of the layer 1 is set in the middle of the left side of the layer 1, and the roller 13-3 arranged on the side of the layer 2 is set In the middle position on the left side of the layer plate 2, the roller 13-3 arranged on the side of the layer plate 3 is set on the left side of the right side of the layer plate 3; when the layers share the same guide rail 13-2, the roller 13-3 The positions on the sides of each layer are the same, for example, they are set in the middle position of the left and right sides of the layer.

As an example, each layer board shares the same guide rail 13-2, and the roller 13-3 is set in the middle position between the left and right sides of the layer board. The layer board is marked as 10, and its structure diagram is shown in Figure 3; Yes, the position of the driving unit 13-1 can be set according to actual needs, for example, set at the bottom of the sales container.

Therefore, after the lift module 13 controls the movement of the floor to be moved, the distance between the camera in the first target storage layer and the product can be enlarged, and the image collected by the camera at this time will be more complete than before the floor is moved. And clear, and based on the image collected by the camera at this time, it will be more accurate to identify the purchased product. That is, in one or more embodiments of this specification, the main control module 12, after the lifting module 13 controls the movement of the floor to be moved, also acquires a fourth image of the first target storage layer, where the fourth image includes the first target Plane image information of the storage layer; and, determine the purchased product according to the first image and the fourth image corresponding to the first target storage layer, and send the product information of the purchased product to the corresponding consumer.

The lifting module 13 also controls the layer to be moved to return to the position before the movement.

Specifically, after determining the movement information of the floor to be moved, the main control module 12 records the movement information of the floor to be moved, and when receiving the movement completion signal sent by the lifting module 13, it sends the second signal according to the receiving time of the movement completion signal. Three image acquisition requests are given to the camera 11 arranged in the first target storage layer, so that the camera 11 in the first target storage layer receives the third image acquisition request according to the receiving time included in the third image acquisition request , Acquire the corresponding video frame in the video stream it shoots, and extract the fourth image from the acquired video frame; when the main control module 12 receives the fourth image sent by the camera 11 in the first target storage layer, it will The first image and the fourth image corresponding to a target storage layer are input into the pre-trained deep neural network for learning, and the product information of the purchased product is obtained. The product information such as product name, product category, product price, etc. The product information of the purchased product is sent to the corresponding consumer, and the recovery information of the floor to be moved is determined according to the recorded movement information of the floor to be moved, and the floor recovery instruction is sent to the lifting module 13 according to the recovery information to make the lifting The module 13 controls the layer to be moved to be restored to the position before the movement according to the restoration information. Alternatively, when the lifting module 13 receives the laminate moving instruction sent by the main control module 12, it records the moving information of the laminate to be moved; after the main control module 12 determines the purchased product, the main control module 12 sends the laminate recovery instruction to Lifting module 13; when the lifting module 13 receives the laminate restoration instruction, it determines the laminate restoration information according to the recorded laminate movement information, and controls the laminate to be moved to the position before the movement according to the laminate restoration information.

The process of acquiring the fourth image by the camera 11 is similar to the process of acquiring the second image by the aforementioned camera 11, which can be referred to the foregoing related description, and the repetitions will not be repeated here. As a result, the purchased product is determined based on the fourth image obtained after moving the laminate, which improves the accuracy of the determination result. By restoring the laminate to the position before the move, it is ensured that when the subsequent consumer purchases the product, every The storage layer has a certain space to facilitate the operation of consumers.

The shelf control equipment provided by one or more embodiments of this specification is provided with a lifting device connected to each shelf. The equipment realizes the flexible control of the position of the shelf in the sales container based on the image collected by the camera, and can not only improve storage The utilization rate of the object space can increase the distance between the camera in the storage layer where the product is operated and the product, reduce the blind spots and blind spots of the camera, and improve the accuracy of the images collected by the camera. Identify the accuracy of the recognition result of the purchased product.

Corresponding to the layer control device described in FIGS. 2 to 3, based on the same technical concept, one or more embodiments of this specification also provide a layer control method applied to a sales container, and the storage space inside the sales container It is divided into multiple storage layers by the slab. FIG. 4 is a schematic flow chart of a method for regulating and controlling laminates according to one or more embodiments of this specification. As shown in FIG. 4, the method includes the following steps.

Step S102: Obtain a first image of each storage layer when the door of the sales container is opened, and a second image and a third image of each storage layer when the door is closed; wherein the first image, the second image, and the The third image is obtained by the camera set on the top side of each storage layer to collect the goods placed in the storage layer where it is located; the first image and the second image include the plane image information of the storage layer, and the third image includes the storage layer. Depth image information of the object layer.

In order to subsequently determine the movement information of the laminate, the camera is a 3D camera, and each video frame in the video stream captured by it includes an image containing plane image information and an image containing depth image information; where the depth image information is like a three-dimensional space Location, size and other information.

Step S104, according to the acquired first image and second image, determine the first target storage layer where the operated commodity is located.

Among them, the operated goods include goods that have been moved by consumers and goods that have been purchased by consumers.

Step S106, according to the third image and the first target storage layer, determine the to-be-moved floor in the floor and the movement information of the to-be-moved floor.

Step S108, controlling the movement of the laminate to be moved according to the movement information of the laminate to be moved.

In one or more embodiments of this specification, based on the images collected by each camera in the sales container, the flexible adjustment of the position of the shelf in the sales container is realized, which can not only improve the utilization rate of storage space, but also increase the number of commodities to be operated. The distance between the camera in the storage layer and the product, thereby reducing the blind spots and blind spots of the camera, improving the accuracy of the image collected by the camera, and improving the accuracy of the recognition result of identifying the purchased product.

Since the operated product is recognized based on the images captured by the camera before and after the consumer purchases the product, that is, the recognition is based on the image captured by the camera before the door of the selling container is opened and after the door is closed; Before and after the product is purchased, the status of the product has changed due to some malicious operations (such as shaking the selling container, etc.) or other factors, so that the problem of the operated product cannot be correctly identified. In one or more embodiments of this specification, the steps Obtaining the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed in S102 includes the following steps.

Step A1: When the door opening signal of the door of the selling container is detected, the first image acquisition request is sent to each camera according to the first detection time of the door opening signal, so that each camera can capture the video according to the first detection time The corresponding video frame is obtained from the stream, and the first image is extracted from the obtained video frame.

Wherein, when the camera receives the first image acquisition request, it acquires the previous video frame of the video frame corresponding to the first detection time in the captured video stream, and extracts the first image from the acquired video frame; or, when the camera receives When it comes to the first image acquisition request, use the first detection time as the cut-off time of the first preset duration, determine the start time corresponding to the first preset duration, and obtain the start time from the captured video stream The corresponding video frame is to extract the first image from the acquired video frame; or, when the camera receives the first image acquisition request, the first detection time is used as the expiration time of the first preset duration, and the captured video stream Any one of the video frames within the first preset period of time is acquired in, and the first image is extracted from the acquired video frame. It should be pointed out that when the camera extracts the first image from the acquired video frame, it can also extract the fifth image containing the depth image information of the storage layer at the same time.

Step A2, when the door closing signal of the selling container is detected, the second image acquisition request is sent to each camera according to the second detection time of the door closing signal, so that each camera can record the video according to the second detection time The corresponding video frame is obtained from the stream, and the second image and the third image are extracted from the obtained video frame.

Wherein, when the camera receives the second image acquisition request, it acquires the next video frame of the video frame corresponding to the second detection time from the captured video stream, and extracts the second image and the third image from the acquired video frame; or When the camera receives the second image acquisition request, the second detection time is used as the start time of the second preset duration, the deadline corresponding to the second preset duration is determined, and the captured video stream is acquired For the video frame corresponding to the deadline, extract the second image and the third image from the acquired video frame; or, when the camera receives the second image acquisition request, use the second detection time as the start of the second preset duration Time, acquire any video frame within the second preset duration in the captured video stream, and extract the second image and the third image from the acquired video frame; or, when the camera receives the second image acquisition request, Taking the second detection time as the start time of the second preset duration, acquire any two video frames within the second preset duration from the captured video stream, and extract the second image from one of the acquired video frames, The third image is extracted from another acquired video frame.

The above-mentioned first preset duration and second preset duration are both very short. The first preset duration and the second preset duration can be the same or different. They can be set according to actual needs; as an example, the first The first preset duration and the second preset duration are the same, and both are 1 second. Thus, by acquiring the first image, the second image, and the third image collected by each camera, the time interval between the collection time of the first image collected by the camera and the first detection time is very short, and the second image is collected by the camera The time interval between the acquisition time and the second detection time is very short. Within this short time interval, it is difficult for consumers or other users to perform malicious operations. Therefore, it can greatly ensure that the first image and the second image are Accuracy, thereby providing an effective data basis for the subsequent determination of the first target storage layer where the operated product is located, the determination of the movement information of the floor to be moved, and the determination of the purchased product.

Since the first image and the second image contain the plane information of the product placed on each storage layer before and after the consumer purchases the product, the first image and the second image corresponding to each storage layer can be processed. Obtain the first target storage layer where the product operated by the consumer is located. Specifically, as shown in FIG. 5, step S104 includes the following.

Step S104-2: Determine the first image and the second image corresponding to each storage layer.

In order to ensure the effective acquisition of images and distinguish the images collected by each camera, in one or more embodiments of this specification, before step S102, it further includes: establishing a transmission channel with each camera to obtain the corresponding camera location through the transmission channel. The acquired image; and establishing a corresponding relationship between the channel identification of the transmission channel and the storage layer identification of the storage layer where the corresponding camera is located; correspondingly, step S104-2 includes: according to the channel identification of the transmission channel through which the first image is acquired, Obtain the corresponding storage layer identifier in the corresponding relationship between the channel identifier and the storage layer identifier, and use the corresponding first image as the first image of the storage layer corresponding to the acquired storage layer identifier; and according to the acquisition of the second image The corresponding storage layer identification is obtained from the corresponding relationship between the channel identification and the storage layer identification, and the corresponding second image is used as the second image of the storage layer corresponding to the acquired storage layer identification .

Or, in order to distinguish the images collected by each camera and distinguish the cameras in each storage layer, in one or more embodiments of this specification, each storage layer is assigned a storage layer identifier to establish a camera The corresponding relationship between the identifier and the storage layer identifier, and the image names of the first image, the second image, and the third image include the camera identifier of the camera that collected the image. Correspondingly, step S104-2 includes: according to each For the camera identification included in the image name of the first image, the corresponding storage layer identification is obtained from the corresponding relationship between the camera identification and the storage layer identification, and the corresponding first image is used as the storage corresponding to the acquired storage layer identification The first image of the layer; and, according to the camera identification included in the image name of each second image, the corresponding storage layer identification is obtained from the corresponding relationship between the camera identification and the storage layer identification, and the corresponding second image is taken as The acquired second image of the storage layer corresponding to the storage layer identifier.

Step S104-4: Perform differential processing on the first image and the second image corresponding to each storage layer to obtain a differential image corresponding to each storage layer.

Specifically, the pixel values of the same pixels of the first image and the second image corresponding to each storage layer are subtracted to obtain the pixel value difference of the corresponding pixel; the pixel value difference is regarded as the corresponding pixel of the difference image The pixel value of the point to obtain the difference image corresponding to each storage layer.

Through differential processing of the first image and the second image corresponding to each storage layer, the similar parts of the first image and the second image corresponding to each storage layer are weakened, and the first image and the second image can be obtained. The difference between the difference image.

Step S104-6: Binarize the difference image corresponding to each storage layer to obtain a binarized image corresponding to each storage layer.

Specifically, corresponding to the pixel points of the differential image corresponding to each storage layer, a matrix including multiple elements to be filled is constructed; the pixel value of each pixel of the differential image of each storage layer is compared with the preset pixel The value threshold is compared. If the pixel value of the pixel of the differential image is greater than the preset pixel value threshold, the corresponding element to be filled in the corresponding matrix is filled with the first preset value; if the pixel value of the pixel of the differential image Not greater than the preset pixel value threshold, fill the corresponding element to be filled in the corresponding matrix with the second preset value; use the image corresponding to the matrix in which all the elements to be filled are filled as the binarization of the corresponding storage layer image. The first preset value is, for example, 1, and the second preset value is, for example, 0. A schematic diagram of a matrix in which all the elements to be filled are filled is shown in FIG. 6.

By binarizing the difference image, a binarized image with only black and white visual effects is obtained, and the interference of other noises is removed, so that according to the binarized image, the first item where the operated product is located can be accurately determined. Target storage layer.

Step S104-8: Determine the target binarized image that meets the preset condition in the binarized image.

Specifically, the number of the first preset values included in the matrix of each binarized image is counted, and if the counted number is greater than the first preset number, then the corresponding binarized image is determined to be the target binarized image; or Count the number of second preset values included in the matrix of each binarized image, and if the counted number is greater than the second preset number, then determine that the corresponding binarized image is the target binarized image. Among them, the first number and the second number may be the same or different, and they can be set according to actual needs.

In step S104-10, the storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.

Since consumers may have moved multiple products in different storage layers or purchased multiple products in different storage layers when purchasing products, the first target storage layer may be one or multiple ; For example, after the consumer picks up the orange juice in storage layer 1 and compares the Coke in storage layer 3, then puts the Coke back into storage layer 2, then storage layer 1, storage layer 2 and storage layer 3 are the first target storage layer; another example, the consumer picked up three bottles of coffee of different brands in storage layer 2, and finally bought two of them, and put the other bottle back in storage layer 2. , The second storage layer is the first target storage layer.

In order to expand the space of the first target storage layer and expand the distance between the camera in the first target storage layer and the goods, so that the camera in the first target storage layer can collect clear and complete images, first Determine the moving information of the floor to be moved and the moving information of the floor to be moved. Specifically, as shown in FIG. 7, step S106 includes the following

S106-2: Determine a third image corresponding to each storage layer.

The implementation method of this step is similar to the process of determining the first image and the second image corresponding to each storage layer in the foregoing step S104-2, and may refer to the foregoing related description, which will not be repeated here.

S106-4: Among the commodities included in the third image corresponding to each storage layer, obtain the target commodity with the smallest vertical distance between the top of the commodity and the top of the storage layer where the commodity is located.

Specifically, in the third image corresponding to each storage layer, measure the vertical distance between the top of each commodity included in it and the top of the storage layer where the commodity is located, and compare the measured vertical distance to obtain the minimum vertical distance , And use the product corresponding to the minimum vertical distance as the target product.

In S106-6, the vertical distance between the top of the target product and the top of the storage layer is taken as the remaining height of the storage layer where the target product is located.

S106-8, according to the remaining height of the storage layer, determine the to-be-moved floor in the floor and the movement information of the to-be-moved floor.

In practical applications, according to the remaining layer height of the storage layer, all the layers with moving space can be determined and used as the layer to be moved, so as to maximize the space of the first target storage layer, so that the first target storage layer can be maximized. The distance between the camera in the object layer and the product is maximized, so that the camera can collect clear and complete images; according to the remaining layer height of the storage layer, part of the layer plate with moving space can be used as waiting Move the shelf so that the distance between the camera in the first target storage layer and the product reaches the preset distance that can collect clear and complete images, without moving too much shelf to ensure safety and stability. Specifically, in one or more embodiments of this specification, as shown in FIG. 8, step S106-8 includes the following steps.

Step B2, if it is determined that there is a storage layer with a non-zero movable distance based on the remaining height of each storage layer except the first target storage layer and the preset safety distance, then according to the non-zero movable distance For the storage layer, determine the layer to be moved.

Specifically, the remaining layer height of each storage layer except the first target storage layer is subtracted from the preset safety distance to obtain the subtraction result, and the subtraction result is used as the movable distance of the corresponding storage layer to determine Whether there is a non-zero movable distance, if so, it is determined that there is a storage layer with a non-zero movable distance; and, according to the number of storage layers with a non-zero movable distance, the number of the first target storage layer, and The positional relationship between the storage layer with a non-zero movable distance and the first target storage layer determines the layer to be moved.

Further, according to the number of storage layers with a non-zero movable distance, the number of the first target storage layer, and the positional relationship between the storage layer with a non-zero movable distance and the first target storage layer, determine the to-be-moved The layer includes: if the number of storage layers with a non-zero movable distance is one, the number of the first target storage layer is one, and the storage layer with a non-zero movable distance and the first target storage layer Adjacent and located above/below the first target storage layer, the bottom/top side layer of the storage layer with a non-zero movable distance is determined to be the layer to be moved; for ease of description, follow the top-to-bottom In order, mark each layer as layer 1, layer 2, layer 3, etc. For example, the storage layer with non-zero movable distance is storage layer 2, and the first target storage layer is storage layer 3. It is determined that the bottom layer of the storage layer 2 is the layer to be moved, that is, the layer 2 is the layer to be moved.

If the number of storage layers with a non-zero movable distance is not unique, the number of the first target storage layer is one, and the storage layer with a non-zero movable distance is continuous and corresponds to the first target storage layer. The storage layer adjacent to and located above/below the first target storage layer is determined from the bottom/top side layer of the storage layer with non-zero movable distance to the top/bottom side layer of the first target storage layer The plate is the layer to be moved; for example, the storage layers with non-zero movable distance are the storage layer 4 and the storage layer 5, and the first target storage layer is the storage layer 3, then the top of the storage layer 5 is determined The bottom layer from the side layer to the storage layer 3 is the layer to be moved, that is, the layer 4 and the layer 3 are the layer to be moved.

If the number of storage layers with non-zero movable distance is not unique, the number of the first target storage layer is one, and the storage layer with non-zero movable distance is a discontinuous storage layer, then when there is a non-zero storage layer When the storage layers with zero movable distance are located above/below the first target storage layer, the bottom/top layer of the storage layer with non-zero movable distance located at the top/bottom will be transferred to the first target storage layer The top/bottom side of the layer is determined to be the layer to be moved; when the storage layer with non-zero movable distance is scattered on the upper and lower sides of the first target storage layer, it will be located at the top of the first target storage layer The bottom side deck of the storage layer with a non-zero movable distance to the top side deck of the first target storage layer, and the storage layer with a non-zero movable distance located at the bottom of the first target storage layer From the top side deck of the first target storage layer to the bottom side deck of the first target storage layer is determined to be the layer to be moved; it should be pointed out that when the number of storage layers with a non-zero movable distance is two, and they are scattered in the first When the upper and lower sides of a target storage layer are both adjacent to the first target storage layer, the bottom layer of the storage layer with non-zero movable distance located at the top of the first target storage layer and the first The top layer of the target storage layer is the same layer, the top layer of the storage layer with non-zero movable distance located at the bottom of the first target storage layer and the bottom side of the first target storage layer The laminate is the same laminate.

For example, if the storage layers with non-zero movable distance are storage layer 1 and storage layer 3, and the first target storage layer is storage layer 5, then the bottom layer of storage layer 1 is plated to storage layer 5. The top layer is determined as the layer to be moved, that is, the layer 1, the layer 2, the layer 3, and the layer 4 are the layer to be moved; another example, the storage layer with a non-zero movable distance is the storage layer 1. Storage layer 2, storage layer 5, the first target storage layer is storage layer 3, then the bottom layer of storage layer 1 to the top side of storage layer 3, and the storage layer The top side layer of 5 to the bottom side of the storage layer 3 is determined to be a movable layer, that is, the layer 1, the layer 2, the layer 3, and the layer 4 are the layers to be moved; The storage layers with zero movable distance are storage layer 2 and storage layer 4, and the first target storage layer is storage layer 3, then the bottom side layer of storage layer 2 and the top side layer of storage layer 4 The board is a movable laminate, that is, the laminate 2 and the laminate 3 are the laminates to be moved.

Furthermore, when the number of the first target storage layer is greater than one, one of the first target storage layers can be used as the current target storage layer in turn, and the movable layer of the current target storage layer can be determined according to the above method. board. It is also possible to maximize the space of each first target storage layer at the same time as the goal, and calculate based on the movable distance of the storage layer with non-zero movable distance and the preset distance, so as to determine the movable layer according to the result of the calculation. For example, the first target storage layer is storage layer 2 and storage layer 5, the storage layers with non-zero movable distance are storage layer 1 and storage layer 4, and the movable storage layer 1 The distance is 9 cm, the movable distance of storage layer 4 is 2 cm, and the preset distance is 5 cm; it can be determined that the bottom layer of storage layer 1 moves up by 5 cm to make the target storage layer 2 reach the preset At this time, storage layer 1 still has a movable distance of 3 cm, and storage layer 5 cannot reach the preset distance after the bottom side of storage layer 4 moves up 2 cm. The bottom shelf of layer 1 moves up 8 cm, the top shelf and bottom shelf of storage layer 3 move up 3 cm, and the bottom shelf of storage layer 4 moves up 3+2=5 cm , The storage layer 5 can reach the shooting distance; alternatively, the bottom layer of the storage layer 1 can be moved up by 9 cm, and the top and bottom layers of the storage layer 3 can be moved up by 4 cm. And the bottom layer of the storage layer 4 moves up 4+2=6 cm, so that the movable distance of the target storage layer 5 is 6 cm, which exceeds the preset distance; therefore, the bottom layer of the storage layer 1 The top and bottom layers of the storage layer 3, and the bottom layer of the storage layer 4 are determined to be the layers to be moved, that is, the layer 1, the layer 2, the layer 3, and the layer 4 are the layers to be moved. Move the shelf. It should be understood that when the number of the first target storage layer is greater than one, the strategy for determining the movable layer can be configured by oneself, which can maximize the space of the target storage layer, and each determines the movable layer. The strategies are all within the scope of protection of this application, and no examples are given here.

Step B4: Determine the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer.

Specifically, when the number of the first target storage layer is one, if the layer to be moved is located above the first target storage layer, the moving direction of the layer to be moved is upward; if the layer to be moved is located at the first target Below the storage layer, the moving direction of the layer to be moved is downward. When the number of the first target storage layer is greater than one, the moving direction of the layer to be moved can be determined according to a specific calculation process.

Step B6: Determine the actual moving distance of the floor to be moved according to the movable distance of the storage layer corresponding to the floor to be moved.

Specifically, if the number of the first target storage layer is one and the number of the layer to be moved is one, the movable distance of the corresponding storage layer is determined to be the actual moving distance of the layer to be moved; for example, The target storage layer is storage layer 3, the layer to be moved is layer 2, and the movable distance of storage layer 2 corresponding to layer 2 is 1.5 cm, then the actual moving distance of the layer 2 to be moved is determined to be 1.5 cm .

If the number of the first target storage layer is one, the number of the layer to be moved is not unique and it is located above/below the target storage layer, each layer to be moved will be the bottom in order from top to bottom/bottom to top. /The movable distance of the storage layer of the top side layer is accumulated, and the accumulated result is used as the actual moving distance of the corresponding movable layer; for example, the first target storage layer is storage layer 4, and the layer to be moved is layer Plate 1, layer 2, and layer 3. The movable distance of storage layer 1 with layer 1 as the bottom layer is 1 cm, and the movable distance of storage layer 2 with layer 2 as the bottom layer The distance is 1.5 cm, and the movable distance of storage layer 3 with layer 3 as the bottom layer is 1 cm, then the actual moving distance of layer 1 is determined to be 1 cm, and the actual moving distance of layer 2 is 1+ 1.5=2.5cm, the actual moving distance of the layer 3 is 1+1.5+1=3.5cm; for another example, the first target storage layer is storage layer 2, and the layer to be moved is layer 2, layer 3 and Layer 4, the movable distance of storage layer 3 with layer 2 as the top side layer is 1 cm, and the movable distance of storage layer 4 with layer 3 as the top side layer is 0 cm. The movable distance of the storage layer 5 where the board 4 is the top side laminate is 2 cm, then it is determined that the actual movement distance of the laminate 4 is 2 cm, and the actual movement distance of the laminate 3 is 2+0=2 cm. The actual moving distance of 2 is 2+0+1=3 cm.

Further, when the number of the first target storage layer is greater than one, the moving distance of the layer to be moved can be determined according to a specific calculation process.

Step B8: Use the moving direction and the actual moving distance of the layer to be moved as the moving information of the layer to be moved.

Through the method described in step B2 to step B8 above, it is possible to determine all the layers with moving space according to the storage layer with non-zero movable distance and use them as the layer to be moved, so as to make the space of the first target storage layer Maximize.

In one or more embodiments of the present specification, on the basis of making the distance between the camera in the first target storage layer and the commodity reach the preset distance that can collect clear and complete images, it is not necessary to move too many layers. To ensure safety and stability, it is also possible to use only part of the laminates with moving spaces as movable laminates; specifically, as shown in FIG. 9, step S106-8 includes the following.

Step C2, if it is determined that there is a distance that is closest to the first target storage layer and the total movable distance is not less than the preset safety distance according to the remaining height of each storage layer except the first target storage layer and the preset safety distance According to the second target storage layer of the distance, the layer to be moved is determined according to the second target storage layer.

Specifically, the remaining layer height of each storage layer except the first target storage layer is subtracted from the preset safety distance, and the subtraction result is used as the movable distance of the corresponding storage layer; and, the current calculation layer The number is counted as one, and it is judged whether the movable distance of a storage layer adjacent to the first target storage layer upward or downward based on the first target storage layer is not less than the preset distance, if yes, determine the one The storage layer is the target storage layer. If not, add one to the current calculated layer number as the current calculated layer number, and use the first target storage layer as the basis to move upward and/or downward, and set the distance from the first target storage layer to The movable distances of the two closest storage layers are added together, and the result of the addition is used as the total movable distance of the two storage layers to determine whether the total distance is not less than the preset distance. Each storage layer is used as the second target storage layer. According to the second target storage layer, determine the layer to be moved; if not, continue to update the current calculated layer number and perform the above calculations until the second target storage layer is obtained.

Further, according to the second target storage layer, determining the layer to be moved includes the following.

If the number of the first target storage layer is one and the second target storage layer is located above/below the first target storage layer, it will be located on the bottom/top side of the top/bottom second target storage layer The top/bottom layer from the layer to the first target storage layer is determined to be the layer to be moved; for example, the first target storage layer is storage layer 4, and the second target storage layer is storage layer 2. And the storage layer 3, the bottom side layer of the storage layer 2 to the top side layer of the storage layer 4 are determined as the layer to be moved, that is, the layer 2 and the layer 3 are the layer to be moved.

If the number of the first target storage layer is one and the second target storage layer is scattered above and below the first target storage layer, then for the second target storage layer located above the first target storage layer, change The bottom side deck of the second target storage layer located at the top to the top side deck of the first target storage layer serves as the floor to be moved; and, for the second target storage layer located below the first target storage layer Layer, the top side layer of the second target storage layer located at the bottom to the bottom side layer of the first target storage layer is used as the layer to be moved; for example, the first target storage layer is storage layer 3. The second target storage layer is storage layer 2, storage layer 4, and storage layer 5. The bottom side slab of storage layer 2 and the top side slab of storage layer 4 and storage layer 5 are determined as The layers to be moved, that is, the layer 2, the layer 3, and the layer 4, are the layers to be moved.

Furthermore, if the number of the first target storage layer is greater than one, one of the first target storage layers can be used as the current target storage layer in turn, and the mobility of the current target storage layer can be determined according to the above method. Laminate. It is also possible to maximize the space of each first target storage layer at the same time as the goal, and calculate based on the movable distance of the storage layer with non-zero movable distance and the preset distance, so as to determine the movable layer according to the result of the calculation. board. It should be understood that when the number of the first target storage layer is greater than one, the strategy for determining the movable layer can be configured by oneself, which can maximize the space of the target storage layer, and each determines the movable layer. The strategies are all within the scope of protection of this application, and no examples are given here.

Step C4: Determine the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer.

For the implementation method of this step, please refer to the relevant description in the foregoing step B4, and the repetitive points will not be repeated here.

Step C6, according to the movable distance of the second target storage layer, determine the actual moving distance of the layer to be moved.

For the implementation method of this step, please refer to the relevant description in the foregoing step B6, and the repetitive points will not be repeated here.

Step C8: Use the moving direction and the actual moving distance of the layer to be moved as the moving information of the layer to be moved.

Through the method of step C2 to step C8, the distance between the camera in the first target storage layer and the product can reach the preset distance that can collect clear and complete images without moving too many layers to ensure safety And stability.

In the above, by determining the movement information of the to-be-moved laminate, the subsequent precise control of the to-be-moved laminate is performed based on the movement information, without damaging part of the goods due to the different heights of goods in the same storage layer.

After the movement information of the floor to be moved is determined, the movement of the floor to be moved can be controlled to maximize the space of the first target storage layer. Specifically, in one or more embodiments of this specification, step S108 Including: according to the moving information of the shelf to be moved and the shelf identification, sending the moving instruction of the shelf to the lifting module set in the sales container, so that the driving unit in the lifting module drives according to the moving information included in the moving instruction of the shelf The layer plate to be moved corresponding to the layer plate identifier moves along the guide rails in the lifting module through the rollers provided on the side of the layer plate.

Further, considering that when the space of the first target storage layer is maximized, the image collected by the camera will be more complete and clearer than before the layer moves, and based on the image collected by the camera at this time, it is determined to be purchased Commodities will also be more accurate. Based on this, in one or more embodiments of this specification, as shown in FIG. 10, after step S108, the following steps are further included.

Step S110: Obtain a fourth image of the first target storage layer, where the fourth image includes planar image information of the first target storage layer.

Specifically, when the movement completion signal sent by the lifting module in the sales container is received, the third image acquisition request is sent to the camera set in the first target storage layer according to the receiving time of the movement completion signal; the first target storage The camera in the layer receives the third image acquisition request, according to the receiving time included in the third image acquisition request, acquires the corresponding video frame in the video stream it shoots, extracts the fourth image from the acquired video frame and returns it; receives the first The fourth image returned by the camera in the target storage layer.

Step S112: Determine the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer.

Specifically, the first image and the fourth image corresponding to the first target storage layer are input into a pre-trained deep neural network for learning, and the product information of the product to be taken out is obtained. The product information such as product name, product category, Commodity prices, etc.

It should be pointed out that the training process of the deep neural network is a technical means well known to those skilled in the art. The specific training process will not be repeated here, and the training process of the existing deep neural network can be referred to.

In step S114, the product information of the purchased product is sent to the corresponding consumer, and the floor to be moved is controlled to be restored to the position before the movement.

Specifically, the product information of the purchased product obtained through learning is sent to the corresponding consumer, so that the consumer can confirm and pay the corresponding fee;

As a result, the purchased product is determined based on the fourth image obtained after moving the laminate, which improves the accuracy of the determination result. By restoring the laminate to the position before the move, it is ensured that when the subsequent consumer purchases the product, every The storage layer has a certain space to facilitate the operation of consumers.

Further, in order to control the laminate to be moved back to the position before the movement, in one or more embodiments of this specification, after step S106-8, it further includes: recording the determined movement information of the laminate to be moved; correspondingly, In step S114, controlling the recovery of the shelf to be moved to the position before the movement includes: determining the recovery information of the shelf to be moved according to the recorded movement information of the shelf to be moved, and sending the recovery instruction of the shelf to the sales container according to the recovery information. The lifting module, so that the lifting module controls the to-be-moved layer board to be restored to the position before the movement according to the restoration information.

Or, when the lifting module in the sales container receives the instruction to move the shelf, it records the movement information of the shelf to be moved; correspondingly, in step S114, controlling the shelf to be moved to be restored to the position before the movement includes: sending the shelf to restore The instruction is given to the lifting module in the sales container, so that the lifting module determines the recovery information of the floor according to the recorded floor movement information, and controls the floor to be moved to be restored to the position before the movement according to the recovery information of the floor.

The shelf adjustment method provided by one or more embodiments of this specification can determine the movement information of the shelf to be moved based on the images collected by each camera in the sales container; it realizes the flexible adjustment of the position of the shelf in the sales container, not only Improve the utilization of storage space, and increase the distance between the camera in the storage layer where the product is operated and the product, thereby reducing the blind spots and blind spots of the camera, and improving the accuracy of the image collected by the camera , And improve the accuracy of the recognition results that identify the purchased goods.

Corresponding to the layer control method described in FIGS. 4 to 10, based on the same technical concept, one or more embodiments of this specification also provide a layer control device applied to a sales container, wherein the storage inside the sales container The space is divided into multiple storage layers by slabs. FIG. 11 is a schematic diagram of the module composition of the laminate control device provided by one or more embodiments of this specification. The device is used to implement the laminate control method described in FIG. 4 to FIG. 10. As shown in FIG. 11, the device includes the following modules .

The acquiring module 201, which acquires the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, The first image, the second image, and the third image are all obtained by a camera arranged on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image And the second image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer.

The first determining module 202 determines the first target storage layer where the operated commodity is located according to the first image and the second image.

The second determining module 203 determines the to-be-moved floor in the floor and the movement information of the to-be-moved floor according to the third image and the first target storage layer.

The control module 204 controls the movement of the laminate to be moved according to the movement information of the laminate to be moved.

Optionally, the acquisition module 201, when detecting a door opening signal that the door of the sales container is opened, sends a first image acquisition request to each of the cameras according to the first detection time of the door opening signal to Make each of the cameras obtain the corresponding video frame in the captured video stream according to the first detection time, and extract the first image from the obtained video frame; and, when it is detected that the door of the sales container is When the door-closing signal is closed, a second image acquisition request is sent to each camera according to the second detection time of the door-closing signal, so that each camera obtains the corresponding image in the captured video stream according to the second detection time. The second image and the third image are extracted from the acquired video frame.

Optionally, the first determining module 202 determines the first image and the second image corresponding to each storage layer; and, for the first image and the second image corresponding to each storage layer The second image is subjected to differential processing to obtain a differential image corresponding to each storage layer; the differential image corresponding to each storage layer is binarized to obtain a binary image corresponding to each storage layer; In the binarized image, a target binarized image that meets a preset condition is determined; the storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.

Optionally, the second determining module 203 determines the third image corresponding to each storage layer; and, among the commodities included in the third image corresponding to each storage layer, obtains the commodity The target product with the smallest vertical distance between the top and the top of the storage layer where the product is located; the vertical distance between the top of the target product and the top of the storage layer where the target product is located is taken as the storage layer where the target product is located. The remaining layer height of the material layer; according to the remaining layer height of the storage layer, determine the layer to be moved in the layer and the movement information of the layer to be moved.

Optionally, the second determining module 203 determines that there is a storage with a non-zero movable distance based on the remaining layer height and a preset safety distance of each storage layer except the first target storage layer. For the object layer, determine the layer to be moved according to the positional relationship between the storage layer with a non-zero movable distance and the first target storage layer; and, according to the positional relationship between the storage layer to be moved and the first target storage layer; The position relationship of the target storage layer determines the moving direction of the layer to be moved; the actual moving distance of the layer to be moved is determined according to the movable distance of the storage layer corresponding to the layer to be moved; The moving direction and actual moving distance of the to-be-moved floor are used as the moving information of the to-be-moved floor; or, if it is based on the remaining floor height and pre-determined height of each storage layer except for the first target storage layer Set a safe distance, determine that there is a second target storage layer that is closest to the first target storage layer and whose total movable distance is not less than a preset distance, then determine the second target storage layer to be moved based on the second target storage layer Layer; and, according to the position relationship between the layer to be moved and the first target storage layer, determine the moving direction of the layer to be moved; according to the movable distance of the second target storage layer, Determine the actual moving distance of the to-be-moved laminate; use the moving direction and the actual moving distance of the to-be-moved laminate as the movement information of the to-be-moved laminate.

Optionally, the control module 204 sends a floor movement instruction to the lifting module set in the sales container according to the moving information and the floor identifier of the floor to be moved, so that the lifting module According to the movement information, the driving unit drives the layer plate to be moved corresponding to the layer plate identifier to move along the guide rail in the lifting module through the roller provided on the side of the layer plate.

Optionally, the acquisition module 201 further acquires a fourth image corresponding to the first target storage layer after the control module 204 controls the movement of the floor to be moved, wherein the fourth image Including the plane image information of the first target storage layer; and, determining the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer; The commodity information of the commodity is sent to the corresponding consumer; the control module 204 also controls the to-be-moved laminate to be restored to the position before the movement.

The layer control device provided in one or more embodiments of this specification can determine the movement information of the layer to be moved based on the images collected by each camera in the sales container; it realizes the flexible control of the position of the layer in the sales container, not only It can improve the utilization of storage space, and can increase the distance between the camera in the storage layer where the product is operated and the product, thereby reducing the blind spots and dead spots of the camera, and improving the accuracy of the image collected by the camera And improve the accuracy of the recognition results of the purchased goods.

It should be noted that the embodiment of the laminate control device in this specification and the embodiment of the laminate control method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the corresponding laminate control method mentioned above. Implementation, the repetition will not be repeated.

Further, corresponding to the methods shown in FIGS. 4 to 10, based on the same technical concept, one or more embodiments of this specification also provide a laminate control device, which is used to execute the foregoing laminate control method, FIG. 12 is a schematic structural diagram of a laminate control device provided by one or more embodiments of this specification.

As shown in Figure 12, the layer control device may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and a memory 302, and the memory 302 may store one or more storage application programs. Or data. Among them, the memory 302 may be short-term storage or persistent storage. The application program stored in the memory 302 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the layer control device. Furthermore, the processor 301 may be configured to communicate with the memory 302, and execute a series of computer-executable instructions in the memory 302 on the layer control device. The layer control device may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input and output interfaces 305, one or more keyboards 306, and so on.

In a specific embodiment, the layer control device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and the one or more programs may include one or more modules, and Each module may include a series of computer-executable instructions for the layer control device, and the one or more programs configured to be executed by one or more processors include computer-executable instructions for performing the following.

Acquire the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first image , The second image and the third image are both obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the second image The image includes the plane image information of the storage layer, the third image includes the depth image information of the storage layer; according to the first image and the second image, determine the first target where the operated commodity is located Storage layer; according to the third image and the first target storage layer, determine the layer to be moved in the layer and the moving information of the layer to be moved; according to the layer to be moved The movement information controls the movement of the layer to be moved.

In one or more embodiments of this specification, based on the images collected by each camera in the sales container, the flexible adjustment of the position of the shelf in the sales container is realized, which can not only improve the utilization of storage space, but also increase the location of the goods to be operated. The distance between the camera in the storage layer and the goods, thereby reducing the blind spots and blind spots of the camera, improving the accuracy of the images collected by the camera, and improving the accuracy of the recognition results for identifying the purchased goods.

Optionally, when the computer-executable instructions are executed, the acquisition of the first image of each storage layer when the door of the sales container is opened, and the first image of each storage layer when the door is closed The second image and the third image include: when the door opening signal that the door of the sales container is opened is detected, a first image acquisition request is sent to each of the cameras according to the first detection time of the door opening signal to Make each of the cameras obtain the corresponding video frame in the captured video stream according to the first detection time, and extract the first image from the obtained video frame; when it is detected that the door of the sales container is closed When the door is closed, a second image acquisition request is sent to each camera according to the second detection time of the door close signal, so that each camera obtains the corresponding video from the captured video stream according to the second detection time Frame, extracting the second image and the third image from the acquired video frame.

Optionally, when the computer-executable instructions are executed, the determining, based on the first image and the second image, the first target storage layer where the operated commodity is located includes: determining that each storage layer corresponds to The first image and the second image of each storage layer; perform differential processing on the first image and the second image corresponding to each storage layer to obtain a differential image corresponding to each storage layer; Binarize the difference image corresponding to the storage layer to obtain the binarized image corresponding to each storage layer; determine the target binarized image that meets the preset conditions in the binarized image; The storage layer corresponding to the binarized image serves as the first target storage layer where the operated commodity is located.

Optionally, when the computer-executable instructions are executed, the third image and the first target storage layer are used to determine the layer to be moved in the layer and the size of the layer to be moved. The movement information includes: determining the third image corresponding to each storage layer; among the commodities included in the third image corresponding to each storage layer, obtaining the difference between the top of the commodity and the top of the storage layer where the commodity is located. The target product with the smallest vertical distance between the target product; the vertical distance between the top of the target product and the top of the storage layer where the target product is located is taken as the remaining height of the storage layer where the target product is located; according to the The remaining layer height of the storage layer is used to determine the layer to be moved in the layer and the movement information of the layer to be moved.

Optionally, when the computer-executable instructions are executed, the determining the floor to be moved in the floor and the movement information of the floor to be moved according to the remaining height of the storage floor includes: If it is determined that there is a storage layer with a non-zero movable distance according to the remaining layer height and the preset safety distance of each storage layer except the first target storage layer, then according to the said storage layer with non-zero movable distance Determine the position relationship between the storage layer and the first target storage layer of the distance, determine the layer to be moved; determine the layer to be moved according to the position relationship between the layer to be moved and the first target storage layer The moving direction of the board; according to the movable distance of the storage layer corresponding to the to-be-moved floor, determine the actual moving distance of the to-be-moved floor; take the moving direction and the actual moving distance of the to-be-moved floor as The movement information of the floor to be moved; or, if it is determined that there is a distance from the first target storage layer according to the remaining height of each storage layer except the first target storage layer and a preset safety distance If the second target storage layer with the closest storey and the total movable distance is not less than the preset distance, the layer to be moved is determined according to the second target storage layer; and the layer to be moved is determined according to the layer to be moved and the second target storage layer. The position relationship of a target storage layer determines the moving direction of the layer to be moved; the actual moving distance of the layer to be moved is determined according to the movable distance of the second target storage layer; The moving direction and the actual moving distance of the moving laminate are used as the moving information of the to-be-moved laminate.

Optionally, when the computer-executable instructions are executed, the controlling the movement of the floor to be moved according to the movement information of the floor to be moved includes: according to the movement information of the floor to be moved and the floor Identification, sending a floor moving instruction to the lifting module set in the sales container, so that the driving unit in the lifting module drives the floor to be moved corresponding to the floor identification to pass through it according to the movement information The rollers arranged on the side move along the guide rails in the lifting module.

Optionally, when the computer-executable instructions are executed, after controlling the movement of the layer to be moved, the method further includes: acquiring a fourth image corresponding to the first target storage layer, wherein the fourth image Including the plane image information of the first target storage layer; determine the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer; The product information is sent to the corresponding consumer, and the laminate to be moved is controlled to be restored to the position before the movement.

The layer control equipment provided by one or more embodiments of this specification can determine the movement information of the layer to be moved based on the images collected by each camera in the selling container; it realizes the flexible control of the position of the layer in the selling container, which can not only improve The utilization of storage space can increase the distance between the camera and the product in the storage layer where the product is operated, thereby reducing the blind spots and dead spots of the camera, and improving the accuracy of the image collected by the camera. And to improve the accuracy of the recognition results for identifying the purchased goods.

It should be noted that the embodiment of the laminate control device in this specification and the embodiment of the laminate control method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the corresponding laminate control method described above. Implementation, the repetition will not be repeated.

Further, corresponding to the methods shown in FIGS. 4 to 10, based on the same technical concept, one or more embodiments of this specification also provide a storage medium for storing computer-executable instructions. A specific embodiment Here, the storage medium may be a U disk, an optical disk, a hard disk, etc., and the computer executable instructions stored in the storage medium can implement the following process when executed by the processor.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, the acquisition of the first image of each storage layer when the door of the sales container is opened, and when the door is closed The second image and the third image of each of the storage layers include: when the door opening signal of the selling container is detected, sending a first image acquisition request according to the first detection time of the door opening signal To each of the cameras so that each of the cameras obtains the corresponding video frame in the captured video stream according to the first detection time, and extracts the first image from the obtained video frame; when the sale is detected When the door of the container is closed with a door-closing signal, a second image acquisition request is sent to each camera according to the second detection time of the door-closing signal, so that each camera can capture the image according to the second detection time. The corresponding video frame is acquired from the video stream, and the second image and the third image are extracted from the acquired video frame.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, the determining the first target storage layer where the operated commodity is located according to the first image and the second image includes: Determine the first image and the second image corresponding to each storage layer; perform differential processing on the first image and the second image corresponding to each storage layer to obtain the corresponding storage layer Binarize the difference image corresponding to each storage layer to obtain the binarized image corresponding to each storage layer; determine the target binary value that meets the preset conditions in the binarized image The target binarized image; the storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, the third image and the first target storage layer are used to determine the layer to be moved in the layer and The movement information of the floor to be moved includes: determining the third image corresponding to each storage layer; among the commodities included in the third image corresponding to each storage layer, acquiring the top of the commodity and the commodity The target product with the smallest vertical distance between the top of the storage layer; the vertical distance between the top of the target product and the top of the storage layer where the target product is located is taken as the value of the storage layer where the target product is located The remaining floor height; according to the remaining floor height of the storage layer, determine the floor to be moved in the floor and the movement information of the floor to be moved.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, the layer to be moved and the layer to be moved in the layer are determined according to the remaining height of the storage layer The movement information of the board includes: if it is determined that there is a storage layer with a non-zero movable distance according to the remaining layer height and the preset safety distance of each storage layer except the first target storage layer, then Determine the position relationship between the storage layer with non-zero movable distance and the first target storage layer to determine the layer to be moved; according to the position relationship between the layer to be moved and the first target storage layer , Determine the moving direction of the floor to be moved; determine the actual moving distance of the floor to be moved according to the movable distance of the storage layer corresponding to the floor to be moved; The moving direction and the actual moving distance are used as the moving information of the layer to be moved; or, if the existing distance is determined based on the remaining layer height of each storage layer except the first target storage layer and the preset safety distance If the first target storage layer is the closest to the second target storage layer whose total movable distance is not less than the preset distance, the layer to be moved is determined according to the second target storage layer; The positional relationship between the moving layer and the first target storage layer determines the moving direction of the layer to be moved; according to the movable distance of the second target storage layer, the actual position of the layer to be moved is determined Moving distance: The moving direction and actual moving distance of the layer to be moved are used as the moving information of the layer to be moved.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, controlling the movement of the layer to be moved according to the movement information of the layer to be moved includes: according to the layer to be moved The movement information of the board and the shelf identification, and the shelf movement instruction is sent to the lifting module set in the sales container, so that the driving unit in the lifting module drives the corresponding to the shelf identification according to the movement information The to-be-moved layer board moves along the guide rail in the lifting module through the rollers provided on the side of the layer.

Optionally, when the computer-executable instructions stored in the storage medium are executed by the processor, after controlling the movement of the floor to be moved, the method further includes: acquiring a fourth image corresponding to the first target storage layer; Wherein, the fourth image includes plane image information of the first target storage layer; determining the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer; The product information of the purchased product is sent to the corresponding consumer, and the laminate to be moved is controlled to be restored to the position before the movement.

When the computer executable instructions stored in the storage medium provided by one or more embodiments of this specification are executed by the processor, they can determine the movement information of the shelf to be moved based on the images collected by the cameras in the sales container; the sales container is realized The flexible adjustment of the position of the middle board can not only improve the utilization of storage space, but also increase the distance between the camera and the product in the storage layer where the product is operated, thereby reducing the blind spot and shooting corner of the camera, and improving The accuracy of the image collected by the camera and the accuracy of the recognition result for identifying the purchased product are improved.

It should be noted that the embodiment of the storage medium in this specification and the embodiment of the layer control method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding layer control method mentioned above. The repetition will not be repeated.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than in the embodiments and still achieve desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown in order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 1930s, the improvement of a technology can be clearly distinguished between hardware improvements (for example, improvements in circuit structures such as diodes, transistors, switches, etc.) and software improvements (improvements in method flow). However, with the development of technology, the improvement of many methods and processes of today can be regarded as a direct improvement of the hardware circuit structure. Designers almost always get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by the hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (such as a Field Programmable Gate Array (Field Programmable Gate Array, FPGA)) is such an integrated circuit whose logic function is determined by the user's programming of the device. It is programmed by the designer to "integrate" a digital system on a PLD, without requiring the chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly realized with "logic compiler" software, which is similar to the software compiler used in program development and writing, but before compilation The original code must also be written in a specific programming language, which is called Hardware Description Language (HDL), and there is not only one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language) , AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description), etc., currently most commonly used It is VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that only a little logic programming of the method flow in the above-mentioned hardware description languages and programming into an integrated circuit can easily obtain the hardware circuit that implements the logic method flow.

The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) executable by the (micro)processor. , Logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicon Labs C8051F320, the memory controller can also be implemented as part of the memory control logic. Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code manner, it is entirely possible to program the method steps to make the controller use logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded logic. The same function can be realized in the form of a microcontroller or the like. Therefore, such a controller can be regarded as a hardware component, and the devices included in it for realizing various functions can also be regarded as a structure within the hardware component. Or even, the device for realizing various functions can be regarded as both a software module for realizing the method and a structure within a hardware component.

The systems, devices, modules, or units explained in the above embodiments may be implemented by computer chips or entities, or implemented by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cell phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or Any combination of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described separately. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same one or more software and/or hardware.

Those skilled in the art should understand that one or more embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

This specification is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in a computer readable medium, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or equipment including a series of elements not only includes those elements, but also includes Other elements that are not explicitly listed, or they also include elements inherent to such processes, methods, commodities, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, commodity, or equipment that includes the element.

One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification can also be practiced in distributed computing environments. In these distributed computing environments, tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

The foregoing descriptions are only examples of the present application, and are not used to limit the present application. For those skilled in the art, this application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the scope of the claims of this application.

Claims

A layer control device applied to a sales container, wherein the storage space inside the sales container is divided into a plurality of storage layers by a layer, and the equipment includes:

A camera arranged on the top side of each of the storage layers; the camera collects a first image of the storage layer where it is when the door of the sales container is opened, and when the door of the sales container is closed Collect the second image and the third image of the storage layer where oneself is at time; wherein, the first image and the second image include the plane image information of the storage layer, and the third image includes the storage layer. Depth image information of the layer;

The main control module, which determines the first target storage layer where the operated commodity is located according to the first image and the second image collected by each of the cameras; and, according to the first target storage layer collected by each of the cameras A third image and the first target storage layer, determining the layer to be moved in the layer plate and the movement information of the layer to be moved;

The lifting module is connected with each of the floor boards, and controls the movement of the floor boards to be moved according to the movement information of the floor boards to be moved.
The device according to claim 1,

The main control module sends a first image acquisition request to each camera according to the first detection time of the door opening signal when detecting the door opening signal that the door of the sales container is opened; and, when detecting Sending a second image acquisition request to each camera according to the second detection time of the door closing signal when the door of the selling container is closed when the door is closed;

The camera, upon receiving the first image acquisition request, acquires a corresponding video frame in the captured video stream according to the first detection time, and extracts the first image from the acquired video frame; and, When the second image acquisition request is received, a corresponding video frame is acquired from the captured video stream according to the second detection time, and the second image and the third image are extracted from the acquired video frame.
The device according to claim 1,

The main control module, which determines the first image and the second image corresponding to each storage layer; and,

Performing differential processing on the first image and the second image corresponding to each storage layer to obtain a differential image corresponding to each storage layer;

Binarize the difference image corresponding to each storage layer to obtain the binarized image corresponding to each storage layer;

Determining a target binarized image that meets a preset condition among the binarized images;

The storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.
The device according to claim 1,

The main control module, which determines the third image corresponding to each storage layer; and,

Among the commodities included in the third image corresponding to each storage layer, obtain the target commodity with the smallest vertical distance between the top of the commodity and the top of the storage layer where the commodity is located;

Taking the vertical distance between the top of the target product and the top of the storage layer where the target product is located as the remaining height of the storage layer where the target product is located;

According to the remaining layer height of the storage layer, determine the layer to be moved in the layer and the movement information of the layer to be moved.
The device according to claim 4,

If the main control module determines that there is a storage layer with a non-zero movable distance based on the remaining layer height and the preset safety distance of each storage layer except the first target storage layer, then according to the Determining the positional relationship between the storage layer having a non-zero movable distance and the first target storage layer, and determining the layer to be moved; and,

Determining the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer;

Determine the actual moving distance of the floor to be moved according to the movable distance of the storage layer corresponding to the floor to be moved;

Using the moving direction and the actual moving distance of the to-be-moved laminate as the movement information of the to-be-moved laminate;

or,

If according to the remaining height of each storage layer except the first target storage layer and the preset safety distance, it is determined that the total movable distance that is closest to the first target storage layer is not less than the preset safety distance. Set the distance to the second target storage layer, then determine the layer to be moved based on the second target storage layer; and,

Determining the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer;

Determine the actual moving distance of the floor to be moved according to the movable distance of the second target storage layer;

The moving direction and the actual moving distance of the to-be-moved laminate are used as the movement information of the to-be-moved laminate.
The device according to claim 1, wherein the lifting module comprises: a driving unit, a guide rail and a roller, wherein the roller is arranged on the side of each of the layer boards;

The main control module sends a layer movement instruction to the drive unit according to the movement information and the layer identifier of the layer to be moved;

The drive unit has an output terminal connected to each of the laminates, and upon receiving the laminate moving instruction, drives the laminate to be moved corresponding to the laminate identifier through the side of the laminate to be moved according to the movement information. The roller moves along the guide rail.
The device according to any one of claims 1-6,

The main control module acquires a fourth image of the first target storage layer after the lifting module controls the movement of the floor to be moved, wherein the fourth image includes the first target storage The plane image information of the layer; and,

Determining the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer;

Sending the product information of the purchased product to the corresponding consumer;

The lifting module also controls the layer to be moved to return to the position before the movement.
A layer control method applied to a sales container, wherein the storage space inside the sales container is divided into a plurality of storage layers by the layer, and the method includes:

Acquire the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first image , The second image and the third image are both obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the second image The image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer;

Determine the first target storage layer where the operated commodity is located according to the first image and the second image;

Determine, according to the third image and the first target storage layer, the to-be-moved floor in the floor and the movement information of the to-be-moved floor;

According to the movement information of the floor to be moved, the movement of the floor to be moved is controlled.
The method according to claim 8, said acquiring the first image of each storage layer when the door of the sales container is opened, and the second image and the second image of each storage layer when the door is closed. The third image includes:

When the door opening signal that the door of the sales container is opened is detected, a first image acquisition request is sent to each camera according to the first detection time of the door opening signal, so that each camera is based on the first detection time. The detection time acquires a corresponding video frame in the captured video stream, and extracts the first image from the acquired video frame;

When the door closing signal of the selling container is detected, the second image acquisition request is sent to each camera according to the second detection time of the door closing signal, so that each camera is based on the second detection time. At the detection time, a corresponding video frame is acquired from the captured video stream, and the second image and the third image are extracted from the acquired video frame.
The method according to claim 8, wherein the determining the first target storage layer where the operated commodity is located according to the first image and the second image comprises:

Determining the first image and the second image corresponding to each storage layer;

Performing differential processing on the first image and the second image corresponding to each storage layer to obtain a differential image corresponding to each storage layer;

Binarize the difference image corresponding to each storage layer to obtain the binarized image corresponding to each storage layer;

Determining a target binarized image that meets a preset condition among the binarized images;

The storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.
The method according to claim 8, wherein the determining the layer to be moved in the layer plate and the movement information of the layer to be moved according to the third image and the first target storage layer includes :

Determining the third image corresponding to each storage layer;

Among the commodities included in the third image corresponding to each storage layer, obtain the target commodity with the smallest vertical distance between the top of the commodity and the top of the storage layer where the commodity is located;

Taking the vertical distance between the top of the target product and the top of the storage layer where the target product is located as the remaining height of the storage layer where the target product is located;

According to the remaining layer height of the storage layer, determine the to-be-moved layer in the layer and the movement information of the to-be-moved layer.
The method according to claim 11, wherein the determining the floor to be moved in the floor and the movement information of the floor to be moved according to the remaining floor height of the storage layer includes:

If it is determined that there is a storage layer with a non-zero movable distance according to the remaining layer height and the preset safety distance of each storage layer except the first target storage layer, then according to the said storage layer with non-zero movable distance The positional relationship between the storage layer of the distance and the first target storage layer, and determine the layer to be moved;

Determining the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer;

Determine the actual moving distance of the floor to be moved according to the movable distance of the storage layer corresponding to the floor to be moved;

Using the moving direction and the actual moving distance of the to-be-moved laminate as the movement information of the to-be-moved laminate;

or,

If according to the remaining height of each storage layer except the first target storage layer and the preset safety distance, it is determined that the total movable distance that is closest to the first target storage layer is not less than the preset safety distance. Set the second target storage layer with a distance, then determine the layer to be moved according to the second target storage layer;

Determining the moving direction of the floor to be moved according to the positional relationship between the floor to be moved and the first target storage layer;

Determine the actual moving distance of the floor to be moved according to the movable distance of the second target storage layer;

The moving direction and the actual moving distance of the to-be-moved laminate are used as the movement information of the to-be-moved laminate.
The method according to claim 8, wherein the controlling the movement of the floor to be moved according to the movement information of the floor to be moved comprises:

According to the movement information and the shelf identification of the shelf to be moved, a shelf movement instruction is sent to the lifting module installed in the sales container, so that the driving unit in the lifting module drives the elevator module according to the movement information. The layer plate to be moved corresponding to the layer plate identifier moves along the guide rail in the lifting module through the roller provided on its side.
The method according to any one of claims 8-13, after said controlling the movement of the layer to be moved, further comprising:

Acquiring a fourth image corresponding to the first target storage layer, where the fourth image includes planar image information of the first target storage layer;

Determining the purchased commodity according to the first image and the fourth image corresponding to the first target storage layer;

The product information of the purchased product is sent to the corresponding consumer, and the laminate to be moved is controlled to return to the position before the movement.
A layer control device applied to a sales container, wherein the storage space inside the sales container is divided into a plurality of storage layers by the layer, and the device includes:

The acquiring module acquires the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, The first image, the second image, and the third image are all obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and The second image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer;

A first determining module, which determines the first target storage layer where the operated commodity is located according to the first image and the second image;

A second determining module, which determines the floor to be moved in the floor and the movement information of the floor to be moved according to the third image and the first target storage layer;

A control module, which controls the movement of the laminate to be moved according to the movement information of the laminate to be moved.
The device according to claim 15,

The first determining module determines the first image and the second image corresponding to each storage layer; and,

Performing differential processing on the first image and the second image corresponding to each storage layer to obtain a differential image corresponding to each storage layer;

Binarize the difference image corresponding to each storage layer to obtain the binarized image corresponding to each storage layer;

Determining a target binarized image that meets a preset condition among the binarized images;

The storage layer corresponding to the target binarized image is used as the first target storage layer where the operated commodity is located.
The device according to claim 15,

The second determining module determines the third image corresponding to each storage layer; and,

Among the commodities included in the third image corresponding to each storage layer, obtain the target commodity with the smallest vertical distance between the top of the commodity and the top of the storage layer where the commodity is located;

The vertical distance between the top of the target product and the top of the storage layer is taken as the remaining height of the storage layer where the target product is located;

According to the remaining layer height of the storage layer, determine the to-be-moved layer in the layer and the movement information of the to-be-moved layer.
The device according to any one of claims 15-17,

The control module, according to the movement information and the shelf identification of the shelf to be moved, sends a shelf movement instruction to the lifting module set in the sales container, so that the driving unit in the lifting module is The moving information drives the to-be-moved layer plate corresponding to the layer plate identifier to move along the guide rail in the lifting module through the roller provided on the side of the layer plate.
A laminate control device, including:

Processor; and,

A memory arranged to store computer-executable instructions which, when executed, cause the processor to:

Acquire the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first image , The second image and the third image are both obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the second image The image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer;

Determine the first target storage layer where the operated commodity is located according to the first image and the second image;

Determine, according to the third image and the first target storage layer, the to-be-moved floor in the floor and the movement information of the to-be-moved floor;

According to the movement information of the floor to be moved, the movement of the floor to be moved is controlled.
A storage medium used to store computer-executable instructions that, when executed, implement the following processes:

Acquire the first image of each storage layer when the door of the sales container is opened, and the second image and the third image of each storage layer when the door is closed; wherein, the first image , The second image and the third image are both obtained by a camera set on the top side of each of the storage layers to collect commodities placed in the storage layer where they are located; the first image and the second image The image includes planar image information of the storage layer, and the third image includes depth image information of the storage layer;

Determine the first target storage layer where the operated commodity is located according to the first image and the second image;

Determine, according to the third image and the first target storage layer, the to-be-moved floor in the floor and the movement information of the to-be-moved floor;

According to the movement information of the floor to be moved, the movement of the floor to be moved is controlled.