WO2021233058A1

WO2021233058A1 - Method for monitoring articles on shop shelf, computer and system

Info

Publication number: WO2021233058A1
Application number: PCT/CN2021/088992
Authority: WO
Inventors: 王璟璟; 吴江旭; 胡淼枫; 聂铭君; 马事伟; 张然
Original assignee: 北京沃东天骏信息技术有限公司; 北京京东世纪贸易有限公司
Priority date: 2020-05-22
Filing date: 2021-04-22
Publication date: 2021-11-25
Also published as: CN111783528B; CN111783528A

Abstract

The present disclosure relates to the technical field of computers, and discloses a method for monitoring articles on a shop shelf, a computer and a system. In the method, by acquiring monitoring images of a shop shelf shot at different times, identifying articles in each monitoring image, and detecting changes in each monitoring image at different times, a change to the articles on the shop shelf is determined promptly and accurately according to the identified articles in each monitoring image and the detected changes of each monitoring image at different times, which enables staff to manage the articles on the shop shelf in a prompt and accurate way.

Description

Method, computer and system for monitoring items on shelves

Cross-references to related applications

This application is based on the application with the CN application number 202010443796.3 and the filing date of May 22, 2020, and claims its priority. The disclosure of the CN application is hereby incorporated into this application as a whole.

Technical field

The present disclosure relates to the field of computer technology, and in particular to a method, computer and system for monitoring items on shelves.

Background technique

Many items are placed on the shelves of the shopping mall, and the user selects the items that he wants to buy from the shelves, and he can also put the items he does not want to buy back to the shelves. After the item is taken away by the user, there will be missing items on the shelf, and items need to be supplemented. The staff of the mall will check the items on the shelves, supplement the missing items, and place the items in the wrong position in the correct position.

Summary of the invention

The inventor found that the management method of shelf items based on manual experience often fails to find items that need to be replenished and items placed incorrectly in a timely manner, resulting in poor timeliness and low accuracy in the management of items on the shelf.

The embodiment of the present disclosure proposes a management solution for shelf items based on image monitoring technology. It acquires the monitored images of the shelves taken at different times, identifies the items in each monitored image, and analyzes the changes in each monitored image at different times. For detection, according to the identified items in each monitoring image and the changes in each monitoring image at different times of detection, the changes of the items on the shelf can be determined in a timely and accurate manner, which is convenient for the staff to check the shelf in a timely and accurate manner. The items on the site are managed.

Some embodiments of the present disclosure propose a method for monitoring items on a shelf, including:

Acquire the first monitoring image and the second monitoring image of the shelf taken at different times;

Identifying each item in the first surveillance image;

Identifying each item in the second surveillance image;

Detecting changes in the first monitoring image and the second monitoring image;

Determine the items on the shelf according to the identified items in the first monitoring image and the items in the second monitoring image and the detected changes in the first monitoring image and the second monitoring image The change situation.

In some embodiments, determining the change of items on the shelf includes:

If the identified item at the first location in the first surveillance image is the same as the item at the first location in the second surveillance image, and the image at the first location in the first surveillance image and the first surveillance image 2. There is a change in the image at the first position in the monitoring image, and it is determined that the number of items at the first position on the shelf has changed;

Or, if the identified item at the second location in the first surveillance image is different from the item at the second location in the second surveillance image, determine that the type of the item at the second location on the shelf has changed;

Or, if there is no item at the third position in the earlier-photographed surveillance image in the identified first surveillance image and the second surveillance image, and there is an object at the third position in the later-photographed surveillance image, determine all The third position on the shelf is supplemented with items;

Or, if there is an item at the fourth position in the monitoring image captured earlier in the identified first monitoring image and the second monitoring image, and there is no item at the fourth position in the monitoring image captured later, it is determined that all The article is missing at the fourth position on the shelf.

In some embodiments, determining that the quantity of items at the first position on the shelf has changed includes:

Acquiring the brightness information of the image at the first position in the first monitoring image and the brightness information of the image at the first position in the second monitoring image;

If the brightness of the image at the first position in the monitoring image captured earlier in the first monitoring image and the second monitoring image is greater than the brightness of the image at the first position in the monitoring image captured later, determine the The number of items in the first position on the shelf is reduced;

If the brightness of the image at the first position in the monitoring image captured later in the first monitoring image and the second monitoring image is greater than the brightness of the image at the first position in the monitoring image captured earlier, determine the The number of items in the first position on the shelf increases.

Acquiring the number of feature points of the image at the first position in the first monitoring image and the number of feature points of the image at the first position in the second monitoring image;

If the number of feature points in the image at the first location in the surveillance image captured earlier in the first surveillance image and the second surveillance image is greater than the number of feature points in the image at the first location in the surveillance image captured later , Determining that the number of items at the first position on the shelf is reduced;

If the number of feature points in the image at the first location in the surveillance image captured later in the first surveillance image and the second surveillance image is greater than the number of feature points in the image at the first location in the surveillance image captured earlier , It is determined that the number of items at the first position on the shelf has increased.

In some embodiments, a change detection model is used to detect changes in the first monitoring image and the second monitoring image;

The change detection model includes a convolutional network and a deconvolutional network that are sequentially cascaded,

The convolutional network includes a plurality of processing modules and an intermediate convolutional layer that are sequentially cascaded, and each processing module includes a first convolutional layer and a pooling layer that are sequentially cascaded;

The deconvolution network includes a plurality of deconvolution modules cascaded in sequence, each deconvolution module includes a deconvolution layer and a second convolution layer cascaded in sequence, and the second convolution layer is configured to match all The output of the cascaded deconvolution layer of the second convolution layer and the output superimposed data of the processing module to which the first convolution layer belongs to the same number of channels as the second convolution layer are subjected to convolution processing.

In some embodiments, identifying each item in the first surveillance image or the second surveillance image includes:

Determining the detection frame of the item in the first surveillance image or the second surveillance image;

Comparing the image feature of the detection frame with the image feature of each item in the database;

The corresponding item in the database that matches the image feature of the detection frame is determined as the item in the detection frame.

In some embodiments, determining the detection frame includes: based on the area of the shelf in the calibrated surveillance image and the position of the shelf of the shelf, determining that the first surveillance image or the second surveillance image is located on the shelf of the shelf The detection frame of the item.

In some embodiments, the method further includes: after acquiring the first monitoring image and the second monitoring image, classifying the first monitoring image and the second monitoring image, and when the classification result shows that the first monitoring image and the second monitoring image When the second surveillance image meets a preset requirement, the identification of each item in the first surveillance image and the second surveillance image is executed.

In some embodiments, it further includes:

If the number of items at the first position on the shelf decreases, a reminder that the items at the first position need to be supplemented;

Or, if the number of items at the first position on the shelf increases, a reminder that the items at the first position have been replenished;

Or, if the type of the item at the second position on the shelf changes, a warning is issued that the item at the second position is incorrectly placed;

Or, if an item is replenished at the third position on the shelf, a reminder that the item at the third position has been replenished;

Or, if there are missing items at the fourth position on the shelf, a prompt is issued that the items at the fourth position need to be supplemented.

Some embodiments of the present disclosure propose a computer for monitoring items on a shelf, including: a memory; and a processor coupled to the memory, and the processor is configured to execute any task based on instructions stored in the memory. A method for monitoring items on a shelf according to an embodiment.

Some embodiments of the present disclosure propose a system for monitoring items on a shelf, including: a computer that monitors the items on the shelf; and a camera device configured to periodically take a monitoring image of the shelf and transmit it to the computer.

Some embodiments of the present disclosure propose a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for monitoring items on a shelf described in any of the embodiments are implemented.

Description of the drawings

The following will briefly introduce the drawings that need to be used in the description of the embodiments or related technologies. According to the following detailed description with reference to the accompanying drawings, the present disclosure can be understood more clearly.

Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

Fig. 1 shows a schematic flowchart of a method for monitoring items on a shelf according to some embodiments of the present disclosure.

Fig. 2 shows a schematic diagram of the installation position of the camera of some embodiments of the present disclosure.

Fig. 3 shows a schematic diagram of a change detection model of some embodiments of the present disclosure.

FIG. 4 shows a schematic diagram of a computer monitoring items on a shelf according to some embodiments of the present disclosure.

FIG. 5 shows a schematic diagram of a system for monitoring items on a shelf according to some embodiments of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

As shown in Fig. 1, the method of this embodiment includes: steps 110-140.

In step 110, the camera device periodically captures the monitoring images of the shelves and transmits them to the computer (referred to as the computer) that monitors the items on the shelves.

Some exemplary parameters of the camera device are listed below, but are not limited to the examples cited. For example, the resolution is above 5 million pixels, and 1080 progressive scanning (Progressive scanning). The focal length is, for example, 2.8 mm (millimeters) or 4 mm, and the corresponding shooting distance is between 0.5 m and 1.5 m. The shooting mode, for example, supports RTSP (Real Time Streaming Protocol) streaming, RTMP (Real Time Messaging Protocol) streaming and timing shooting functions. The shooting interval can be set by the shooting time period and shooting frequency, for example, every day from 9:00 am to 9:00 pm, shooting once every set time (such as 10 minutes). The power supply mode may be battery power supply, for example, and the power supply time is more than three months. The network connection mode may be, for example, a wifi (Wireless Fidelity) connection or a mobile communication card (such as a fourth-generation mobile communication card) connection. For example, the installation method can be a separate installation of the camera and the battery. The camera can be attached to the appropriate position of the shelf with a mother-and-child tape, and the battery can be connected to the camera with a cable, or the camera and the battery can be installed in one piece. Below the shelf support plate.

Fig. 2 shows a schematic diagram of the installation position of the camera of some embodiments of the present disclosure. The figure shows two rows of shelves, and the camera device 200 is fixed on the left shelf, and is used to capture the surveillance image of the right shelf.

In step 120, the computer obtains each monitoring image of the shelf captured by the camera device at different times, and recognizes the items in each monitoring image.

For example, for the first monitoring image and the second monitoring image of the shelf taken at different times, the computer recognizes each item in the first monitoring image and recognizes each item in the second monitoring image.

In some embodiments, after acquiring the surveillance image, the computer classifies the surveillance image, and when the classification result indicates that the surveillance image meets the preset requirements, the operation of identifying each item in the surveillance image is performed.

Among them, the computer can use the image classification model to classify the monitored image, for example, it is classified as normal, too bright, too dark, oblique, blurred, and there are obstructing objects in the image, etc., for the classified as normal monitoring image, perform recognition in the monitoring image For other categories, the operation of each item in the surveillance image will not be performed to identify each item in the surveillance image. As a result, it filters out unqualified surveillance images caused by imaging quality (such as images that are too dark, too bright, severely tilted, and blurred), environmental interference (such as shelves in the image being blocked by people, shopping carts, and other objects). Improve the accuracy of item identification.

Among them, the image classification model can be obtained by training convolutional neural networks such as resnet-50 using training images, and the loss function used for training is cross-entropy for multi-classification tasks, and then the convolutional neural network is updated iteratively using the gradient descent method. Until the preset training termination condition is met, such as the iteration reaches a certain number of times, or the change in loss is less than the preset value.

One of the first monitoring image and the second monitoring image is a reference image, and the other is a to-be-detected image. Usually, the monitoring image of the shelf taken when the items on the shelf are displayed in a standard and complete state can be used as the reference map. For example, the operating time of the mall is from 9:00 a.m. to 9:00 p.m., and the monitoring images of the shelves taken when the items on the shelves are displayed standard and complete at 9:00 a.m. can be used as the reference map. The monitoring images of the shelves taken in 10 minutes are used as the images to be inspected.

In some embodiments, the method for the computer to identify each item in the surveillance image (such as the first surveillance image or the second surveillance image, etc.) includes: steps 121-122.

In step 121, the detection frame of the item in the surveillance image is determined.

In the captured surveillance image, there may be other backgrounds besides the shelf, which will cause calculation errors during analysis. Therefore, by calibrating the shelf area, you can determine which areas in the image are shelf images. After the camera is fixed, manually mark the shelf area on the surveillance image, that is, mark the shelf area with a rectangular frame on the surveillance image, and record the coordinates of the rectangular frame. In order to determine the position of the item on the shelf, it is necessary to know the position of the shelf on the shelf. After the camera device is fixed, the position of the shelf partitions is manually calibrated on the surveillance image, that is, the shelf partitions are marked with a straight line segment in the shelf area of the surveillance image, and the end coordinates of the straight line segment are recorded. Based on the area of the shelf in the calibrated surveillance image and the position of the shelf of the shelf, the detection frame of the item located on the shelf of the shelf in the surveillance image is determined.

For example, the monitoring image is input to a single-stage target detection model, such as YOLO (You only look once), SSD (Single Shot MultiBox Detector), etc., and the single-stage target detection model outputs the detection of all items in the monitoring image Box information, such as the position and size of the detection box.

In step 122, the image feature of the detection frame is compared with the image feature of each item in the database, and the corresponding item in the database that matches the image feature of the detection frame is determined as the item in the detection frame.

Collect one or more images of each known item, extract the features of these images using a feature extraction model, and save the identification of the known item and the image features of the known item to the database. The feature extraction model is used to extract the image features of the detection frame. Calculate the distance between the image feature of the detection frame and the image feature of each item in the database. The distance can be, for example, the Euclidean distance. When the minimum distance is less than a preset threshold, the item in the database with the smallest distance from the detection frame will be calculated. Determined as the item in the detection box. Among them, the identifier of the item may be, for example, the inventory keeping unit (SKU) of the item, and the SKU is a unique identifier for each item. Among them, the feature extraction model can be obtained by training convolutional neural networks such as resnet-50 using training images. The training loss function is a triple-center loss (triplet-center loss, TCL) function, that is, triplet loss (triplet loss). ) Function and the center loss (center loss) function, and then use the gradient descent method to iteratively update the parameters of the convolutional neural network until the preset training termination condition is met, such as the iteration reaches a certain number of times, or the change in loss is less than default value. The trained convolutional neural network is the feature extraction model. Input an image into the feature extraction model, and then the corresponding feature of the image can be output. The feature dimension is, for example, 256.

The information stored in the database is shown in the following table:

序号Serial number	SKUSKU	特征向量Feature vector	时间戳Timestamp
00	1024267410242674	[0.32，0.52，0.01…][0.32, 0.52, 0.01...]	2020.02.142020.02.14
11	1024267410242674	[0.12，0.35，0.04…][0.12, 0.35, 0.04...]	2020.02.142020.02.14
22	264951002264951002	[0.08，0.03，0.81…][0.08, 0.03, 0.81...]	2020.01.012020.01.01

In step 130, for the first monitoring image and the second monitoring image of the shelf taken at different times, the computer detects the change of the first monitoring image and the second monitoring image.

In some embodiments, a change detection model is used to detect changes in the first monitoring image and the second monitoring image. The first monitoring image and the second monitoring image are superimposed (or spliced) into the change detection model, and an output image with the same size as the input image is output. The output image represents the first monitoring image and the second monitoring image The image of the change. The changes of the first monitoring image and the second monitoring image include: the image at the same position in the first monitoring image and the second monitoring image does not change (as shown by the blank circle in FIG. 3) and the first monitoring image There is a change in the image at the same position as in the second monitoring image. The image at the same position in the first surveillance image and the second surveillance image has changes that can be further refined as: the image of the item at the same position in the first surveillance image and the second surveillance image disappears (as shown in Figure 3) Dotted circle) or the item image change (as shown by the slashed circle in Figure 3).

Among them, the change detection model includes a convolutional network and a deconvolutional network that are sequentially cascaded; the convolutional network includes a plurality of processing modules and an intermediate convolutional layer that are sequentially cascaded, and each processing module includes a first volume that is sequentially cascaded The deconvolutional layer and the pooling layer; the deconvolutional network includes a plurality of deconvolutional modules cascaded in sequence, and each deconvolutional module includes a deconvolutional layer and a second convolutional layer that are cascaded sequentially. The second The convolutional layer is configured to superimpose the output of the deconvolutional layer cascaded with the second convolutional layer and the output of the processing module to which the first convolutional layer belongs to the same number of channels as the second convolutional layer. , Splicing data) for convolution processing.

Fig. 3 shows a schematic diagram of a change detection model of some embodiments of the present disclosure. The change detection model 300 in FIG. 3 includes 4

processing modules

311, 312, 313, 314, an intermediate convolutional layer 320, and 4

anti-processing modules

331, 332, 333, 334, of which each

processing module

311, 312, 313, 314 include a first convolutional layer and a pooling layer, and each

de-processing module

331, 332, 333, 334 includes a deconvolutional layer and a second convolutional layer. The first convolutional layers in the

processing modules

311, 312, 313, and 314 are all 3×3 convolution kernels, and the number of channels is 64, 128, 256, 512, and the number of channels for the middle convolutional layer 320 is 1024. The second convolutional layers in the

anti-processing modules

331, 332, 333, and 334 are all 3×3 convolution kernels, and the number of channels is 512, 256, 128, and 64 in sequence.

In step 140, the computer determines the items on the shelf according to the identified items in the first surveillance image and the items in the second surveillance image, and the detected changes in the first surveillance image and the second surveillance image. The change situation.

In some embodiments, the method of determining the change of items on the shelf includes:

If the identified item at the first location in the first surveillance image is the same as the item at the first location in the second surveillance image, and the image at the first location in the first surveillance image and the second surveillance image are There is a change in the image at the first position, and it is determined that the number of items at the first position on the shelf has changed;

Or, if there is no item at the third position in the earlier-photographed surveillance image in the identified first surveillance image and the second surveillance image, and there is an item at the third position in the later-photographed surveillance image, it is determined that the shelf is Added items in the third position;

Or, if there is an item at the fourth position in the monitoring image taken earlier in the identified first monitoring image and the second monitoring image, and there is no item at the fourth position in the monitoring image taken later, it is determined that the shelf is Items are missing at the fourth position.

In some embodiments, the method for determining the change in the number of items at the first position on the shelf includes: acquiring brightness information of the image at the first position in the first surveillance image (for example, the first surveillance image in the first surveillance image). The average brightness of the image pixels in the area of a location) and the brightness information of the image at the first location in the second monitoring image (such as the average brightness of the image pixels in the area of the first location in the second monitoring image); if The brightness of the image at the first position in the monitoring image captured earlier in the first monitoring image and the second monitoring image is greater than the brightness of the image at the first position in the monitoring image captured later, and it is determined that the first position on the shelf is The number of items at the location is reduced; if the brightness of the image at the first location in the surveillance image captured later in the first surveillance image and the second surveillance image is greater than the image at the first location in the surveillance image captured earlier The brightness determines the increase in the number of items in the first position on the shelf.

In some other embodiments, the method for determining how the number of items at the first position on the shelf has changed includes: acquiring the number of feature points of the image at the first position in the first surveillance image and the second surveillance image The number of feature points in the image at the first position in the first monitoring image and the second The number of feature points in the image at the first position determines that the number of items at the first position on the shelf decreases; The number of feature points of the image is greater than the number of feature points of the image at the first position in the monitoring image taken earlier, and it is determined that the number of items at the first position on the shelf has increased. Among them, the feature points are, for example, feature points extracted by the ORB (Oriented FAST and Rotated Brief) algorithm. ORB algorithm is a fast feature point extraction and description algorithm. The ORB algorithm is divided into two parts, namely feature point extraction and feature point description. Feature extraction is developed by the FAST (features from accelerated segment test) algorithm. The feature point description is based on the BRIEF (Binary Robust Independent Elementary Features) feature description algorithm. .

If the number of items at the first position on the shelf decreases, a reminder that the items at the first position needs to be supplemented is issued; or if the number of items at the first position on the shelf increases, the items at the first position are sent out. Supplementary reminder; or, if the type of the item at the second position on the shelf changes, a reminder that the item at the second position is incorrectly placed; or, if the item is supplemented at the third position on the shelf, the first A reminder that the items at the third position have been replenished; or, if there are no items at the fourth position on the shelf, a reminder that the items at the fourth position need to be replenished.

For the above example of determining the change of the item on the shelf, refer to the following table, where the change detection refers to the change of the image of the item at the same position in the two monitoring images.

The business meanings corresponding to the changes in the items on the shelves above are as follows:

Suppose the surveillance image at t1 shows that there is an item Sku1 on the shelf at a certain position, and the subsequent surveillance image at t2 shows that there is no item on the shelf at that position, indicating that the items at this location are sold out and you need to replenish items, you can send out Sku1 Need to add hints.

Assuming that the surveillance image at time t1 shows that there is no item at a certain position on the shelf, and the subsequent surveillance image at time t2 shows that the item Sku2 is placed on the shelf at that position, indicating that the item is replenished at this position, a reminder that Sku2 has been replenished can be issued.

Assume that the surveillance images at time t1 and the subsequent time at t2 both show that an item Sku3 is placed on the shelf. However, the surveillance image at time t2 is relative to the surveillance image at time t1. The brightness or/and feature points of the image at that location The quantity is reduced by 20%, which means that part of the item Sku3 has been sold, and part of it has not been sold. At this time, you can send a reminder that Sku3 needs to be supplemented, or you can wait for all Sku3 to be sold out before sending out a reminder that Sku3 needs to be supplemented.

Assume that the surveillance images at time t1 and the subsequent time at t2 both show that the item Sku4 is placed on the shelf. However, the surveillance image at time t2 is relative to the surveillance image at time t1. The brightness or/and feature points of the image at that location A 20% increase in the quantity indicates that the item Sku4 has been supplemented, and a reminder that Sku4 has been supplemented can be issued.

Suppose the surveillance image at t1 shows that the item Sku5 is placed on the shelf at a certain position, and the subsequent surveillance image at t2 shows that the item Sku6 is placed on the shelf at that position, indicating that Sku6 has not been placed back to its original position, and Sku6 is issued Wrong prompt.

Assuming that the surveillance images at time t1 and subsequent time t2 both show that an item Sku7 is placed at a certain position on the shelf, and the change detection finds that the item image has not changed, it means that the Sku7 has always been in the original state and no one has purchased it.

In some embodiments, a small wireless camera device is installed on the opposite side of the shelf, and the shooting angle is aimed at the shelf. After the camera device is fixed, adjust the brightness, resolution, etc. of the captured image. After the adjustment is completed, the area of the shelf is marked with a rectangular frame in the monitoring image, and the partition of the shelf is marked with a straight line. Set the shooting interval, such as shooting and uploading a surveillance image every 10 minutes. Set the working time, such as 8:00～22:00, and sleep in the rest of the time. The computer receives each surveillance image from the camera device, uses the surveillance image taken at 8:00 as the reference map, and uses the surveillance images taken at various time points after 8:00 as the map to be tested, identifies the items in each surveillance image, and Compare the changes of each to-be-detected image with respect to the reference image, and use the strategy described in step 140 to determine the shelf in real time according to the items in each monitored image identified and the changes of each detected image relative to the reference image. And send out corresponding reminders on the changes of the items on the website. An exemplary monitoring data of a shopping mall is, for example:

The above-mentioned management scheme of shelf items based on image monitoring technology realizes that by acquiring the monitoring images of the shelves taken at different times, identifying the items in each monitoring image, and detecting the changes of each monitoring image at different times, according to the recognized The items in each monitoring image and the changes of each monitoring image at different times of detection can timely and accurately determine the changes of the items on the shelf, which is convenient for the staff to manage the items on the shelf in time and accurately.

As shown in FIG. 4, the computer 400 of this embodiment includes a memory 410 and a processor 420 coupled to the memory 410. The processor 420 is configured to execute any of the foregoing embodiments based on instructions stored in the memory 410. Method of monitoring the items on the shelf.

Among them, the memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, and the like. The system memory stores, for example, an operating system, an application program, a boot loader (Boot Loader), and other programs.

The computer 400 may also include an input/output interface 430, a network interface 440, a storage interface 450, and the like. These

interfaces

430, 440, 450 and the memory 410 and the processor 420 may be connected via a bus 460, for example. Among them, the input and output interface 430 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networked devices. The storage interface 450 provides a connection interface for external storage devices such as SD cards and U disks.

As shown in FIG. 5, the system 500 of this embodiment includes a computer 400 that monitors items on a shelf and one or more camera devices 200.

The camera device 200 is configured to periodically take a monitoring image of the shelf and transmit it to the computer 400.

The computer 400 is configured to execute the method for monitoring items on a shelf in any of the foregoing embodiments based on the received monitoring image.

Some embodiments of the present disclosure also propose a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for monitoring items on a shelf in any of the foregoing embodiments are implemented.

Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more non-transitory computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program codes. .

The present disclosure is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are used to generate It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

The above are only the preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection of the present disclosure. Within range.

Claims

A method of monitoring items on shelves, including:

Acquire the first monitoring image and the second monitoring image of the shelf taken at different times;

Identifying each item in the first surveillance image;

Identifying each item in the second surveillance image;

Detecting changes in the first monitoring image and the second monitoring image;

Determine the items on the shelf according to the identified items in the first monitoring image and the items in the second monitoring image and the detected changes in the first monitoring image and the second monitoring image The change situation.
The method according to claim 1, wherein determining the change of items on the shelf comprises:

If the identified item at the first location in the first surveillance image is the same as the item at the first location in the second surveillance image, and the image at the first location in the first surveillance image and the first surveillance image 2. There is a change in the image at the first position in the monitoring image, and it is determined that the number of items at the first position on the shelf has changed;

Or, if the identified item at the second location in the first surveillance image is different from the item at the second location in the second surveillance image, determine that the type of the item at the second location on the shelf has changed;

Or, if there is no item at the third position in the earlier-photographed surveillance image in the identified first surveillance image and the second surveillance image, and there is an object at the third position in the later-photographed surveillance image, determine all The third position on the shelf is supplemented with items;

Or, if there is an item at the fourth position in the monitoring image captured earlier in the identified first monitoring image and the second monitoring image, and there is no item at the fourth position in the monitoring image captured later, it is determined that all The article is missing at the fourth position on the shelf.
The method according to claim 2, wherein determining that the quantity of items at the first position on the shelf has changed comprises:

Acquiring the brightness information of the image at the first position in the first monitoring image and the brightness information of the image at the first position in the second monitoring image;

If the brightness of the image at the first position in the monitoring image captured earlier in the first monitoring image and the second monitoring image is greater than the brightness of the image at the first position in the monitoring image captured later, determine the The number of items in the first position on the shelf is reduced;

If the brightness of the image at the first position in the monitoring image captured later in the first monitoring image and the second monitoring image is greater than the brightness of the image at the first position in the monitoring image captured earlier, determine the The number of items in the first position on the shelf increases.
The method according to claim 2, wherein determining that the quantity of items at the first position on the shelf has changed comprises:

Acquiring the number of feature points of the image at the first position in the first monitoring image and the number of feature points of the image at the first position in the second monitoring image;

If the number of feature points in the image at the first location in the surveillance image captured earlier in the first surveillance image and the second surveillance image is greater than the number of feature points in the image at the first location in the surveillance image captured later , Determining that the number of items at the first position on the shelf is reduced;

If the number of feature points in the image at the first location in the surveillance image captured later in the first surveillance image and the second surveillance image is greater than the number of feature points in the image at the first location in the surveillance image captured earlier , It is determined that the number of items at the first position on the shelf has increased.
The method according to claim 1, wherein a change detection model is used to detect changes in the first monitoring image and the second monitoring image;

The change detection model includes a convolutional network and a deconvolutional network that are sequentially cascaded,

The convolutional network includes a plurality of processing modules and an intermediate convolutional layer that are sequentially cascaded, and each processing module includes a first convolutional layer and a pooling layer that are sequentially cascaded;

The deconvolution network includes a plurality of deconvolution modules cascaded in sequence, each deconvolution module includes a deconvolution layer and a second convolution layer cascaded in sequence, and the second convolution layer is configured to match all The output of the cascaded deconvolution layer of the second convolution layer and the output superimposed data of the processing module to which the first convolution layer belongs to the same number of channels as the second convolution layer are subjected to convolution processing.
The method according to claim 1, wherein identifying each item in the first surveillance image or the second surveillance image comprises:

Determining the detection frame of the item in the first surveillance image or the second surveillance image;

Comparing the image feature of the detection frame with the image feature of each item in the database;

The corresponding item in the database that matches the image feature of the detection frame is determined as the item in the detection frame.
The method according to claim 6, wherein determining the detection frame comprises:

Based on the area of the shelf in the calibrated surveillance image and the position of the shelf of the shelf, the detection frame of the item located on the shelf of the shelf in the first surveillance image or the second surveillance image is determined.
The method according to claim 1, further comprising:

After acquiring the first surveillance image and the second surveillance image, classify the first surveillance image and the second surveillance image, and when the classification result shows that the first surveillance image and the second surveillance image conform to the preset When required, the identification of each item in the first surveillance image and the second surveillance image is executed again.
The method according to claim 2, further comprising:

If the number of items at the first position on the shelf decreases, a reminder that the items at the first position need to be supplemented;

Or, if the number of items at the first position on the shelf increases, a reminder that the items at the first position have been replenished;

Or, if the type of the item at the second position on the shelf changes, a warning is issued that the item at the second position is incorrectly placed;

Or, if an item is replenished at the third position on the shelf, a reminder that the item at the third position has been replenished;

Or, if there are missing items at the fourth position on the shelf, a prompt is issued that the items at the fourth position need to be supplemented.
A computer that monitors the items on the shelf, including:

Memory; and

A processor coupled to the memory, the processor being configured to execute the method for monitoring items on a shelf according to any one of claims 1-9 based on instructions stored in the memory.
A system for monitoring items on shelves, including:

The computer for monitoring the items on the shelf as claimed in claim 10; and

The camera device is configured to periodically take a monitoring image of the shelf and transmit it to the computer.
A non-transitory computer-readable storage medium with a computer program stored thereon, and when the program is executed by a processor, the steps of the method for monitoring items on a shelf according to any one of claims 1-9 are realized.