CN115661624A - Digital method and device for goods shelf and electronic equipment - Google Patents
Digital method and device for goods shelf and electronic equipment Download PDFInfo
- Publication number
- CN115661624A CN115661624A CN202211388161.3A CN202211388161A CN115661624A CN 115661624 A CN115661624 A CN 115661624A CN 202211388161 A CN202211388161 A CN 202211388161A CN 115661624 A CN115661624 A CN 115661624A
- Authority
- CN
- China
- Prior art keywords
- shelf
- commodity
- original image
- target image
- commodities
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 82
- 238000001514 detection method Methods 0.000 claims abstract description 77
- 238000009826 distribution Methods 0.000 claims abstract description 22
- 238000012937 correction Methods 0.000 claims abstract description 14
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 23
- 238000010586 diagram Methods 0.000 claims description 23
- 230000009466 transformation Effects 0.000 claims description 23
- 238000003860 storage Methods 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 10
- 239000000284 extract Substances 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 15
- 238000007726 management method Methods 0.000 description 19
- 238000004891 communication Methods 0.000 description 10
- 230000002265 prevention Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000008269 hand cream Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a digital method of a goods shelf, which comprises the steps of collecting an original image of the goods shelf on which commodities are displayed; carrying out shelf corner point detection and geometric correction on the original image to obtain a first target image; providing the first target image for a commodity detection and recognition model, and extracting a characteristic value of a commodity contained in the first target image; clustering the commodities on the shelf displayed by the first target image according to the characteristic value of the commodities; comparing the information with preset information to obtain the identification result of the commodities placed in each shelf area after clustering; acquiring commodity distribution information of the goods shelf according to the identification result of each commodity and the goods shelf area; in the process, the incidence relation is established between the identification result of the commodity in the first target image and the shelf area, so that the distribution information of the commodity is obtained, and the shelf is digitally managed according to the distribution information of the commodity.
Description
Technical Field
The application relates to the technical field of computer vision, in particular to a digital method and device for a shelf, electronic equipment and a computer readable storage medium.
Background
Intelligent loss prevention services have become one of the hot spots in recent years as the focus of super-daily management work of large retailers. In the intelligent loss prevention business, the digital management of the shelf is used as an important component, and how to realize the digital management of the shelf and further improve the accuracy and the accuracy of human-cargo interaction behaviors becomes a technical problem to be solved urgently.
In the existing digital shelf management technology, a commodity detection model is generally adopted to detect an image of a commodity to be identified, and classification and statistics of shelf commodities are further realized according to judgment of information such as commodity appearance characteristics in the image of the commodity to be identified; however, in the above process, when other shelf commodities exist in the acquired image, the commodity detection model is easy to perform repeated statistics on the commodities, and when a customer interacts with the shelf commodities, the commodity detection model cannot provide accurate goods taking information, which easily causes frequent occurrence of situations such as erroneous judgment and erroneous judgment, and finally causes that the precision and accuracy of commodity statistics in commodity digital management cannot be guaranteed, and the actual scene requirement of the hyperintelligent loss prevention service of each major business cannot be met.
Disclosure of Invention
The embodiment of the application provides a shelf digitalization method and device, electronic equipment and a computer readable storage medium, so as to solve the above problems in the prior art.
The embodiment of the application provides a digital method for a shelf, which comprises the following steps: acquiring an original image of a shelf on which commodities are displayed; carrying out shelf corner point detection and geometric correction on the original image to obtain a first target image; providing the first target image for a commodity detection and identification model, and extracting a characteristic value of a commodity contained in the first target image; clustering the commodities on the shelf displayed by the first target image according to the characteristic value of the commodities; comparing the information with preset information to obtain the identification result of the commodities placed in each shelf area after clustering; and acquiring the commodity distribution information of the shelf according to the identification result of each commodity and the shelf area.
Optionally, the shelf corner point detection and geometric correction include: inputting the original image into a shelf corner detection model to obtain corner coordinates of the shelf in the original image; and carrying out affine transformation on the original image according to the corner point coordinates to obtain the first target image.
Optionally, the inputting the original image into a shelf corner detection model to obtain corner coordinates of the shelf in the original image includes: inputting the original image into the shelf corner detection model to obtain a multi-scale feature corresponding to the original image; fusing the multi-scale features to obtain a thermodynamic diagram corresponding to the original image; and calculating to obtain the corner point coordinates according to the thermodynamic diagram.
Optionally, performing affine transformation on the original image according to the corner coordinates to obtain the first target image, includes: taking the corner coordinates as affine transformation corner coordinates, and calculating to obtain an affine transformation matrix according to the initial corner coordinates and standard rectangular vertex coordinates corresponding to mapping; and carrying out affine transformation on the original image according to the affine transformation matrix to obtain the first target image.
Optionally, the first target image is a front view, and the shelf is a standard rectangle in the first target image.
Optionally, the commodity detection and identification model is a commodity detection, positioning and identification model including a label allocation strategy.
Optionally, the commodity detection and identification model adopts a yolov5 target detection algorithm based on anchor-free detection head and SimOTA label distribution.
Optionally, the comparing with the preset information to obtain the identification result of the commodity placed in each shelf area after clustering includes one of the following modes: comparing the characteristic value of the commodity with the characteristic data of the commodity contained in a prestored commodity information database to obtain the identification result of the commodity placed in each shelf area after clustering; according to the goods shelf coordinate area where the goods are clustered, comparing the goods shelf goods placement planning information provided by the goods shelf display chart according to the placement order to obtain the recognition result of the goods placed in each goods shelf area after clustering; and combining the two modes to obtain the identification result of the commodities placed in each shelf area after the clustering.
Optionally, the method further includes: and updating the commodity distribution information of the goods shelf and the characteristic value of the commodity to a commodity information database.
The embodiment of the present application further provides a digital device for a shelf, the device includes:
an acquisition unit configured to acquire an original image of a shelf on which a commodity is displayed;
the correcting unit is configured to perform shelf corner point detection and geometric correction on the original image to obtain a first target image;
the extraction unit is configured to provide the first target image to a commodity detection and recognition model and extract a characteristic value of a commodity contained in the first target image;
a clustering unit configured to cluster commodities on a shelf displayed by the first target image according to the characteristic value of the commodity;
the comparison unit is configured to compare the preset information to obtain the identification result of the commodities placed in each shelf area after clustering;
and the synthesizing unit is configured to obtain the commodity distribution information of the shelves according to the identification result of each commodity and the shelf area.
The embodiment of the application also provides an electronic device, which comprises a processor and a memory; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method.
Embodiments of the present application also provide a computer-readable storage medium having one or more computer instructions stored thereon, which are executed by a processor to implement the above-mentioned method.
Compared with the prior art, the embodiment of the application has the following advantages:
according to the digital method for the goods shelf, the original image of the goods shelf on which commodities are displayed is collected; carrying out shelf corner point detection and geometric correction on the original image to obtain a first target image; the first target image can reflect the relative position of the commodity on the shelf to which the commodity belongs, and the first target image is provided for a commodity detection and recognition model to extract the characteristic value of the commodity contained in the first target image; clustering the commodities on the shelf displayed by the first target image according to the characteristic value of the commodities; obtaining the identification result of the commodities placed in each shelf area after clustering by comparing with preset information; according to the identification result of each commodity and the area of the shelf, commodity distribution information of the shelf is obtained; the commodity identification result and the goods shelf area are associated to obtain the distribution information of the commodities, and then digital management of the goods shelf is achieved.
Drawings
Fig. 1 is a schematic view of an application scenario of the digital method for shelf provided in the present application.
Fig. 2 is a flow chart of a method for digitizing a shelf provided in a second embodiment of the present application.
Fig. 3 is a schematic diagram of an original image provided in a second embodiment of the present application.
Fig. 4 is a flowchart illustrating a first target image acquisition process provided in a second embodiment of the present application.
Fig. 5 is a schematic diagram of clustering of commodities in a first target image provided in a second embodiment of the present application.
Fig. 6 is a block diagram of a shelf digitizer apparatus according to a third embodiment of the present application.
Fig. 7 is a schematic logical structure diagram of an electronic device according to a fourth embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit and scope of this application, and thus this application is not limited to the specific implementations disclosed below.
First, some technical terms related to the present application are explained:
the core of the intelligent loss prevention service is that stock data of an account and inventory data of physical commodities are checked and corrected in super-daily operation of each merchant, and loss of different commodities at the stage is obtained by obtaining a difference between the stock data of the account and the inventory data of the physical commodities, so that digital management of goods shelves is realized.
And the digital management of the goods shelf is used for expressing the process of converting the characteristic information and the relative position relation information into digital information and storing the digital information in a goods database by acquiring the characteristic information of the physical goods shelf and goods on the goods shelf and the relative position relation information of the goods on the goods shelf.
The commodity code, also called SKU (Stock Keeping Unit) code, is a code classification method after the product is put in storage, and is also the minimum Unit for inventory control. The commodity SKU code can be in units of parts, boxes, trays and the like, each product corresponds to a unique SKU code, the SKU code comprises attributes of a product such as brand, model, configuration, grade, packaging capacity, unit, production date, quality guarantee period, use, price, production place and the like, the attribute of one product is different from that of other products, the commodity is a single product, the physical commodity can be memorized and checked conveniently through the commodity SKU code, and the modern management of the commodity is further realized.
The nature of the anchor-free detection algorithm, as one of the types of target detection algorithms, is to detect a target object by predicting the distance between a target center point and a frame from the center point or the width and height of the target, and typical representatives of the detection algorithms are an anchor-point algorithm and a key-point algorithm.
And (3) allocating the SimOTA labels, wherein the label allocation principle is to regard the label allocation problem as an optimal transmission problem, calculate the transportation cost among all the prediction boxes, and enable the transportation cost to be the lowest by searching a proper mapping relation.
The yolov5 target detection algorithm is a single-stage target detection algorithm, a plurality of new improved ideas are added on the basis of yolov4, yolov5 adopts a lighter network structure, and an FPN enhanced feature extraction network is used for replacing PAN (personal area network), so that the model is simpler, the speed is higher, a rounding method is used for searching the adjacent position, the target is mapped to a plurality of surrounding central grid points, and the speed and the precision are greatly improved.
The HRNet network, also known as a high resolution network, has the feature of maintaining a higher resolution throughout the network operation. The specific process is that high-resolution subnets are gradually increased to low-resolution subnets from the beginning of a first stage, more stages are formed, and the multi-resolution subnets are connected in parallel. In the whole process, multi-scale repeated fusion is carried out by repeatedly exchanging information on parallel multi-resolution sub-networks, and the key points are estimated through high-resolution representation output by the network, and the network can realize accurate prediction of the key points on a predicted thermodynamic diagram.
In order to facilitate understanding of the digitization method of the shelf provided in the embodiment of the present application, before describing the embodiment of the present application, a background of the embodiment of the present application is described.
With the change of digitalization and the change of marketing modes, more and more businesses apply self-service shopping to the daily operation of supermarkets. However, although the self-service shopping method increases the convenience of shopping for customers to a certain extent, the loss rate in the shopping process is increased, and therefore, the self-service shopping presents a new challenge to the intelligent loss prevention business of each large business. The digital management of the goods shelf is used as the key of the intelligent loss prevention service, and how to improve the accuracy and the precision rate in the digital management process of the goods shelf becomes the key problem of the current intelligent loss prevention service.
In the current digital management of goods on shelves, the common way is to detect the image of the goods to be identified based on a static goods detection model, and to query and count the kinds and the quantity of the goods on shelves based on the image of the goods. However, the above method frequently needs to count the number and state of the commodities on the shelf, and the actual count data of the commodities is checked with the inventory data on the account, so that the loss of the current different kinds of commodities is known, and then the current sales condition of the supermarket is known.
With the above background, those skilled in the art can appreciate the problems in the prior art, and in view of this, the present application provides a method for digitizing a shelf, which solves the above problems in the prior art. Next, an application scenario of the shelf digitization method according to the present application will be described in detail. The digital method of the goods shelf provided by the embodiment of the application can be applied to the field of digital management of the goods shelf or other related technical fields with goods shelf management requirements.
First, an application scenario of the shelf digitization method according to the embodiment of the present application will be described below.
Fig. 1 is a schematic application scenario diagram of a shelf digitization method according to a first embodiment of the present application.
As shown in fig. 1, the application scenario includes: a terminal 101 and a server 102; the terminal 101 and the server 102 are connected in communication through a network.
Taking fig. 1 as an example to illustrate in detail, in the application background of super-business daily operation, a terminal 101 located in a super-business, such as a mobile phone terminal of a supermarket manager, or a surveillance video device arranged inside the super-business, collects an original image of a shelf on which commodities are displayed at any time, transmits the collected original image to a server 102 through network communication connection, and after receiving the original image, a server at the server 102 performs corner detection and geometric correction on the shelf on the original image shot at any angle to obtain a first target image; inputting the first target image into a commodity detection and recognition model of a server, and extracting characteristic values of commodities contained in the first target image, such as information of coordinate positions of the commodities, depth characteristic values of the commodities, similarity characteristic values of the commodities and the like; according to the characteristic values, clustering the commodities on the shelf displayed by the first target image; comparing the characteristic value of the commodity with preset information to obtain the identification result of the commodity placed in each shelf area after clustering; the identification result of the commodity is used as a code unique to each commodity and can be used as the identity information of the commodity; and finally, acquiring the commodity distribution information of the shelf according to the identification result of each commodity and the shelf area. Through the process, the placing information of the goods relative to the goods shelf forms corresponding digital information, and the digital information is stored in the goods shelf management system, so that the digital management of the goods shelf is realized.
Fig. 1 is a schematic view of an application scenario of a shelf digitization method provided in an embodiment of the present application, where the embodiment of the present application does not limit the devices included in fig. 1, and does not limit the number of the terminals 101 and the servers 102. For example, in the application scenario shown in fig. 1, the application scenario may further include a data storage device, where the data storage device may be an external memory with respect to the terminal 101 and the server 102, or may be an internal memory integrated in the terminal 101 and the server 102. The terminal 101 may be a smartphone, a smart band, a tablet computer, a wearable device, a multimedia player, an electronic reader, and other devices having a communication function, and an Application (APP) having a photographing function is correspondingly installed on the device; the server 102 may be a server or a cluster of several servers, or may be a cloud computing service center.
In the embodiment of the present application, the number of devices of the terminal 101 and the server 102 in fig. 1 may vary. Those skilled in the art can understand that the application scenario in fig. 1 is only an example of the digital method in the shelf of the present application, and does not constitute a limitation to the method, and the terminal 101 and the server 102 may also be located on the same side of network communication or on different sides of network communication, and the description of the embodiment is merely illustrated as a reference.
The specific implementation process of the application scenario can be referred to the following scheme description of each embodiment.
After the application of the embodiment of the present application is introduced, the present application further provides a shelf digitizing method, and an apparatus, an electronic device, and a computer-readable storage medium corresponding to the method. The following provides embodiments for detailed description of the above method, apparatus, electronic device, and computer-readable storage medium.
The first embodiment of the application provides a digital method for a shelf.
Fig. 2 is a flowchart of a digital method for a shelf according to an embodiment of the present application, and the method according to the embodiment is described in detail below with reference to fig. 2. The following description refers to embodiments for illustrating the principles of the methods and is not meant to be limiting in actual use.
As shown in fig. 2, the method for digitizing a shelf provided in this embodiment includes the following steps:
s201, an original image of a shelf on which a commodity is displayed is acquired.
This step serves to obtain an original image of the shelf on which the product is displayed.
In this embodiment, the original image may be an image of a business scene acquired at any observation angle, and the acquisition mode of the original image may be an artificial shooting mode, a robot fixed-point cruise shooting mode, or a monitoring lens shooting mode. The format of the original image may be various types such as RGB format and JPG format, and this embodiment is not particularly limited.
For convenience of understanding the original image collected in this step, please refer to fig. 3 by way of example, and fig. 3 is a schematic diagram of the original image. In this figure, the shelf on which the product is displayed has a non-standard rectangular frame structure when viewed from the current image, and has a rectangular frame structure when viewed from the current imaging angle, and information about the product on the shelf can be obtained from this angle of view. After the image acquisition equipment of the terminal shoots the original image, the original image is transmitted to a server of a server through network communication, so that the server can conveniently perform subsequent processing.
S202, carrying out shelf corner point detection and geometric correction on the original image to obtain a first target image.
The effect of this step is to obtain a corrected first target image.
It should be understood that, since the shape of the shelf in the original image is not a standard rectangle, which is not beneficial to the detection and identification of the goods on the shelf, and is also not beneficial to presenting the relative position relationship between the goods and the shelf, for example, the goods a is located at the X-th layer of the shelf 15 cm away from the left edge of the shelf, the shelf in the original image needs to be geometrically corrected to obtain the first target image corrected to be a rectangular shelf. Corresponding to the original image, the shelf has corresponding initial corner point coordinates in the original image, which are used to indicate the positions of the four corner points of the shelf in the original image. In the digitization method for the shelf of the present application, the first target image is a front view, and the shelf is a standard rectangle in the first target image.
To facilitate understanding of the process, please refer to the schematic diagram in fig. 4, and fig. 4 is a flowchart illustrating the first target image acquiring process.
The above shelf corner point detection and geometric correction includes: inputting the original image into a shelf corner detection model to obtain corner coordinates of the shelf in the original image; and carrying out affine transformation on the original image according to the corner point coordinates to obtain the first target image.
In this embodiment, the shelf corner detection model is specifically a shelf corner detection model based on an HRNet network, and the original image is input into the shelf corner detection model based on the HRNet network for detection and identification, so that a complete boundary of the shelf can be obtained, and the corner coordinates of the shelf in the original image are obtained through calculation.
Inputting the original image into a shelf corner detection model to obtain corner coordinates of the shelf in the original image, wherein the method comprises the following steps:
inputting the original image into the shelf corner detection model to obtain a multi-scale feature corresponding to the original image; fusing the multi-scale features to obtain a thermodynamic diagram corresponding to the original image; and calculating to obtain the corner coordinates according to the thermodynamic diagram.
The above steps are aimed at obtaining mapped shelf corner coordinates corresponding to the initial corner coordinates. For convenience of understanding, a further description is made, where the shelf corner detection model is a network detection model with different resolutions, the detection model can extract multi-scale features of multiple commodities and shelves in an original image, the scale features have corresponding relations with the commodities and the shelves in the original image, that is, the multi-scale features are used for representing the shape and appearance of a specific commodity, and the multi-scale features are formed by combining multiple data; the method comprises the steps of fusing multi-scale features extracted from an original image into a thermodynamic diagram one fourth of the original image, calculating and obtaining corner coordinates of the shelf according to a response center point in the thermodynamic diagram, wherein the corner coordinates are used as coordinates after initial corner coordinates are mapped.
Performing affine transformation on the original image according to the corner coordinates to obtain the first target image, including:
taking the corner coordinates as affine transformation corner coordinates, and calculating to obtain an affine transformation matrix according to the initial corner coordinates and standard rectangular vertex coordinates corresponding to mapping; and performing affine transformation on the original image according to the affine transformation matrix to obtain the first target image.
In this step, since the angle of view of the image capturing device at the terminal is a point-like divergent angle of view when the shelf is photographed, there is a certain angle of view deviation between the photographed original image of the shelf and the front view of the shelf, it is necessary to perform geometric correction on the original image of the shelf, and map an irregular quadrilateral shelf to a rectangular shelf, so as to accurately calculate and obtain the relative position between the shelf and the product, or the relative coordinates of the product on the shelf, for example, the initial corner coordinates on one side of the top of the shelf are (22, 13, 97), and the vertex coordinates of the corresponding standard rectangle are (52, 75, 64).
Mapping the initial corner coordinates of the shelf in the original image obtained in the previous step to the corresponding standard rectangle vertex coordinates, and calculating to obtain an affine transformation matrix, wherein the affine transformation matrix can be expressed as follows:
on the basis of obtaining the affine transformation matrix, multiplying all pixel points in the original image by the affine transformation matrix to obtain a first target image with a standard rectangular shelf.
Through the steps, the original image is subjected to affine transformation to obtain a first target image, and the shelf in the image is a standard rectangle.
S203, providing the first target image to a commodity detection and identification model, and extracting a characteristic value of a commodity contained in the first target image.
The step is used for acquiring the characteristic value of the commodity in the first target image.
In this embodiment, the above product detection and identification model is a product detection, positioning and identification model including a label allocation policy, and the product detection and identification model employs yolov5 target detection algorithm based on anchor-free detection head and SimOTA label allocation.
In the digitalization method of the goods shelf, the goods shelf is detected and positioned by adopting a yolov5 target detection algorithm based on the allocation of an anchor-free detection head and a SimOTA label, wherein the anchor-free detection head is used as a component of a goods detection and identification model software output module, a goods block diagram and the characteristic types of the goods can be output, the method is different from the anchor-base detection head, the anchor-free detection head does not need to search the size of the anchor in advance, the super-parameter setting is reduced, and meanwhile, the method can better adapt to a dense target detection task. The SimOTA label distribution method can calculate an optimal label distribution strategy by combining classification loss and frame loss, and dynamically updates the number of positive samples in training. Based on the anchor-free detection head and the yolov5 target detection algorithm distributed by the SimOTA label, the finally obtained commodity detection and identification model can achieve the operation effects of high accuracy and high recall rate.
In the shelf digitization method of the present application, the characteristic values of the product include at least: the method comprises the following steps of determining the coordinate position of a commodity, the depth characteristic value of the commodity, the similarity characteristic value of the commodity, the appearance and appearance characteristics of the commodity, the name of the commodity and other characteristics.
In the operation process of the product detection and identification model of this embodiment, the first target image is input to the model, so that the product detection and identification model can identify the product in the image, detect the coordinate position of each product, cut out each product image from the shelf of the first target image according to the coordinate position of each product, and generate a corresponding product block diagram, and the product identification model extracts feature values, such as a depth feature value and a similarity feature value, for representing the product according to the product block diagram. The depth feature value is used to represent a 256-dimensional feature vector of the commodity (the depth feature value is a feature vector extracted by a commodity identification model), and the similarity feature value is used to represent the similarity between the commodity and the commodity in the commodity information database of the commodity detection and identification model. In the process of identifying the commodity detection and identification model, the feature value can be extracted by using the image appearance of the commodity, the feature value can be extracted by using the packaging image and text of the commodity, and the feature value can be extracted by using the two-dimensional code on the image appearance of the commodity.
Through the above steps, the present embodiment obtains a feature value for characterizing the commodity in the first target image.
And S204, clustering the commodities on the shelf displayed by the first target image according to the characteristic value of the commodity.
The step is used for classifying the commodities in the first target image according to the characteristic values of the commodities.
In this embodiment, the clustering manner of the commodities in the first target image may be to cluster the commodities according to depth feature values of the commodities, may also be to cluster the commodities according to similarity feature values of the commodities, may also be to implement clustering of the commodities according to coordinate positions of the commodities, and the like, and the form of performing category division based on the feature values of the commodities is various, which is not specifically limited in this embodiment.
For easy understanding of the above clustering manner, reference may be made to fig. 5, where fig. 5 is a schematic diagram illustrating the clustering of the commodities in the first target image. In the figure, a block diagram of the goods on the shelf acquired from the first target image has a plurality of goods, such as hand cream, paper towel, diaper, and the like, and clustering of the goods on the shelf is realized by extracting feature values related to each of the goods, such as a good, a coordinate position is [ 62,33,15 ], a similarity feature value is "toilet paper 98%", a good, B, a depth feature value 9, a coordinate position is [ 22,9,74 ], a good, a coordinate position is [ 19, 55, 2 ], a depth feature value is 4, and a similarity feature value is "diaper 77%".
Through the steps, the commodity clustering in the first target image is achieved.
S205, comparing with preset information to obtain the identification result of the commodity placed in each shelf area after clustering.
The step has the effect that the commodities on the shelf displayed by the clustered first target image are compared with preset information to obtain the identification result of the placed commodities.
The step includes one of the following ways:
according to the characteristic value of the commodity, comparing the characteristic value with commodity characteristic data contained in a prestored commodity information database to obtain an identification result of the commodity placed in each clustered goods shelf area;
according to the shelf coordinate area where the commodities are clustered, comparing the shelf commodity placement planning information provided by the shelf display chart according to the placement order to obtain the identification result of the commodities placed in each shelf area after clustering;
and combining the two modes to obtain the identification result of the commodities placed in each shelf area after the clustering.
For convenience of understanding, a further description is made, in the commodity detection and identification model, a prestored commodity information database is further provided, in which feature information of a plurality of commodities, such as the arrangement number of the commodities, the categories of the commodities, the SKU numbers of the commodities, and the like, is stored, in the database, a layout drawing of the commodities on shelves is also prestored, and by comparing the feature values of the commodities in the first target image with the commodity feature data included in the commodity information database, the identification result of the clustered commodities in the belonging shelf area can be obtained; or obtaining the identification result of the clustered commodities by combining coordinate information of the clustered commodities on the shelves and a shelf display chart roughly set in advance.
In this embodiment, the identification result of the product refers to a code of the product, i.e., a product SKU code. The characteristic value of the commodity is compared with commodity characteristic data in a prestored commodity information database, or the characteristic value of the commodity is compared with planning and placing information of a shelf display chart according to a shelf coordinate area obtained after the commodity is clustered on a shelf, so that the SKU code of the commodity placed in each area on the shelf can be obtained. The acquisition of the SKU codes can realize one-to-one correspondence between the areas and the positions of the commodities on the goods shelf and the SKU codes of the commodities, so that business management of business overload is facilitated.
Through the steps, the characteristic value of the commodity is compared with the preset information, and the identification result of the commodity is obtained.
And S206, acquiring the commodity distribution information of the shelves according to the identification result of each commodity and the shelf area.
The step has the function of associating the identification result of the commodity with the shelf area where the commodity is located to obtain the distribution information of the corresponding shelf commodity.
In this step, the distribution information of the shelf products can be realized by the SKU code of each product on the shelf and the placement position information of the product on the shelf. For example, the SKU code of the article a is "SKM150GB12T4", the relative position coordinate of the article a on the shelf is "56, 77, 32", the position coordinate can indicate both the relative position of the article a on the shelf and the actual geographical position of the article a in the marketplace, and the distribution information of the article of the category to which the article a belongs in the marketplace can be obtained by correlating the SKU code of the article a with the information of the area of the shelf where the article a is located, such as "the first floor, 15 cm from the left edge of the shelf". In this embodiment, the commodity distribution information of the shelf and the characteristic value of the commodity may be updated to the commodity information database, so as to implement the digitization process of the shelf.
Through the steps, the commodity distribution information of the shelves can be obtained, and the digital management of the shelf commodities can be realized for users.
According to the digital method for the goods shelf, the association relation is established between the commodity identification result and the goods shelf area, the part information of the commodity is obtained, and then digital management of the goods shelf is achieved.
The second embodiment provides a method for digitizing a shelf, and correspondingly, an embodiment of the present application also provides a device for digitizing a shelf, which is relatively simple in description because the device embodiment is substantially similar to the method embodiment, and the details of the related technical features can be found in the corresponding description of the method embodiment provided above, and the following description of the device embodiment is only illustrative.
As shown in fig. 6, a block diagram of a digitizing apparatus for a shelf provided in this embodiment includes:
an acquisition unit 601 configured to acquire an original image of a shelf on which a commodity is displayed;
a correction unit 602 configured to perform shelf corner detection and geometric correction on the original image to obtain a first target image;
an extracting unit 603 configured to provide the first target image to a commodity detection and recognition model, and extract a feature value of a commodity contained in the first target image;
a clustering unit 604 configured to cluster the commodities on the shelf displayed by the first target image according to the characteristic value of the commodity;
a comparison unit 605 configured to compare the preset information with the preset information to obtain the identification result of the commodity placed in each shelf area after the clustering;
the synthesizing unit 606 is configured to obtain the product distribution information of the shelf according to the recognition result of each product and the shelf area where the product is located.
The above embodiments provide a digital device for a shelf, and in addition, embodiments of the present application also provide an electronic device, which is basically similar to the method embodiments and therefore is described relatively simply, and the details of the related technical features need to be referred to the corresponding descriptions of the method embodiments provided above, and the following description of the electronic device embodiments is only illustrative. The embodiment of the electronic equipment is as follows: please refer to fig. 7 for understanding the present embodiment, wherein fig. 7 is a schematic diagram of an electronic device provided in the present embodiment.
As shown in fig. 7, the electronic device provided in this embodiment includes: a processor 701 and memory 702, a communication bus 703, and a communication interface 704. The processor 701 is configured to execute the one or more computer instructions to implement the steps of the above method embodiments. The memory 702 is used to store one or more computer instructions for data processing. The communication bus 703 is used to connect the processor 701 and the memory 702 mounted thereon. The communication interface 704 is configured to provide a connection interface for the processor 701 and the memory 702.
In the embodiments, a shelf digitalizing method, a device and an electronic device corresponding to the method are provided, and in addition, a computer-readable storage medium for implementing the shelf digitalizing method is also provided in the embodiments of the present application. The embodiments of the computer-readable storage medium provided in the present application are described more simply, and in relevant parts, reference may be made to the corresponding descriptions of the above method embodiments, and the embodiments described below are only illustrative.
The present embodiment provides a computer readable storage medium having stored thereon computer instructions, which when executed by a processor, implement the steps of the above-described method embodiments.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
1. Computer-readable media, including both permanent and non-permanent, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
2. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the appended claims.
Claims (12)
1. A method for digitizing a shelf, comprising:
acquiring an original image of a shelf on which commodities are displayed;
carrying out shelf corner point detection and geometric correction on the original image to obtain a first target image;
providing the first target image for a commodity detection and recognition model, and extracting a characteristic value of a commodity contained in the first target image;
clustering the commodities on the shelf displayed by the first target image according to the characteristic value of the commodity;
comparing the information with preset information to obtain the identification result of the commodities placed in each shelf area after clustering;
and obtaining the commodity distribution information of the goods shelf according to the identification result of each commodity and the shelf area.
2. The method of claim 1, wherein the shelf corner point detection and geometric correction comprises:
inputting the original image into a shelf corner detection model to obtain corner coordinates of the shelf in the original image;
and performing affine transformation on the original image according to the corner coordinates to obtain the first target image.
3. The method for digitizing shelf according to claim 2, wherein the inputting the original image into a shelf corner detection model to obtain the coordinates of the corner of the shelf in the original image comprises:
inputting the original image into the shelf corner detection model to obtain a multi-scale feature corresponding to the original image;
fusing the multi-scale features to obtain a thermodynamic diagram corresponding to the original image;
and calculating to obtain the corner point coordinates according to the thermodynamic diagram.
4. The digital shelf method according to claim 2, wherein the affine transformation of the original image according to the corner coordinates to obtain the first target image comprises:
taking the corner coordinates as affine transformation corner coordinates, and calculating to obtain an affine transformation matrix according to the initial corner coordinates and standard rectangular vertex coordinates corresponding to mapping;
and carrying out affine transformation on the original image according to the affine transformation matrix to obtain the first target image.
5. The method of digitizing shelves according to claim 1, wherein the first target image is a front view and the shelf is a standard rectangle in the first target image.
6. The shelf digitization method according to claim 1, wherein the item detection and identification model is an item detection location identification model that contains a label assignment strategy.
7. The method for digitizing shelves of claim 6, wherein the commodity detection and recognition model employs yolov5 target detection algorithm based on Anchor-free detection head and SimOTA label assignment.
8. The shelf digitization method according to claim 1, wherein the comparing with the preset information to obtain the identification result of the goods placed in each shelf area after clustering comprises one of the following ways:
comparing the characteristic value of the commodity with the characteristic data of the commodity contained in a prestored commodity information database to obtain the identification result of the commodity placed in each shelf area after clustering;
according to the shelf coordinate area where the commodities are clustered, comparing the shelf commodity placement planning information provided by the shelf display chart according to the placement order to obtain the identification result of the commodities placed in each shelf area after clustering;
and combining the two modes to obtain the identification result of the commodities placed in each shelf area after the clustering.
9. The method of digitizing shelves of claim 1, further comprising:
and updating the commodity distribution information of the goods shelf and the characteristic value of the commodity to a commodity information database.
10. An apparatus for digitizing a shelf, the apparatus comprising:
an acquisition unit configured to acquire an original image of a shelf on which a commodity is displayed;
the correcting unit is configured to perform shelf corner point detection and geometric correction on the original image to obtain a first target image;
the extraction unit is configured to provide the first target image to a commodity detection and recognition model and extract a characteristic value of a commodity contained in the first target image;
a clustering unit configured to cluster the commodities on the shelf displayed by the first target image according to the characteristic value of the commodity;
the comparison unit is configured to compare the preset information to obtain the identification result of the commodities placed in each shelf area after clustering;
and a synthesizing unit configured to obtain commodity distribution information of the shelf according to the identification result of each commodity and the shelf area.
11. An electronic device comprising a processor and a memory; wherein, the first and the second end of the pipe are connected with each other,
the memory is to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of any one of claims 1-9.
12. A computer-readable storage medium having stored thereon one or more computer instructions for execution by a processor to perform the method of any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211388161.3A CN115661624A (en) | 2022-11-07 | 2022-11-07 | Digital method and device for goods shelf and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211388161.3A CN115661624A (en) | 2022-11-07 | 2022-11-07 | Digital method and device for goods shelf and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115661624A true CN115661624A (en) | 2023-01-31 |
Family
ID=85015203
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211388161.3A Pending CN115661624A (en) | 2022-11-07 | 2022-11-07 | Digital method and device for goods shelf and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115661624A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2805760C1 (en) * | 2023-02-13 | 2023-10-23 | Самсунг Электроникс Ко., Лтд. | Method and device for retail goods identification |
| CN117196779A (en) * | 2023-09-16 | 2023-12-08 | 广州小明网络科技有限公司 | Novel retail management system based on store |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102930264A (en) * | 2012-09-29 | 2013-02-13 | 李炳华 | System and method for acquiring and analyzing commodity display information based on image identification technology |
| CN106650682A (en) * | 2016-12-29 | 2017-05-10 | Tcl集团股份有限公司 | Method and device for face tracking |
| CN110874751A (en) * | 2018-08-29 | 2020-03-10 | 北京京东尚科信息技术有限公司 | Supermarket replenishment method and system |
| CN112464952A (en) * | 2020-11-25 | 2021-03-09 | 创新奇智(南京)科技有限公司 | Image alignment method and image matching identification method |
| CN113674049A (en) * | 2021-06-24 | 2021-11-19 | 杭州碑石数据技术有限公司 | Commodity shelf position identification method and system based on picture search and storage medium |
| CN114494892A (en) * | 2022-04-15 | 2022-05-13 | 广州市玄武无线科技股份有限公司 | Goods shelf commodity display information identification method, device, equipment and storage medium |
| CN114511820A (en) * | 2022-04-14 | 2022-05-17 | 美宜佳控股有限公司 | Goods shelf commodity detection method and device, computer equipment and storage medium |
| CN115249356A (en) * | 2022-09-21 | 2022-10-28 | 浙江莲荷科技有限公司 | Identification method, device, equipment and storage medium |
-
2022
- 2022-11-07 CN CN202211388161.3A patent/CN115661624A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102930264A (en) * | 2012-09-29 | 2013-02-13 | 李炳华 | System and method for acquiring and analyzing commodity display information based on image identification technology |
| CN106650682A (en) * | 2016-12-29 | 2017-05-10 | Tcl集团股份有限公司 | Method and device for face tracking |
| CN110874751A (en) * | 2018-08-29 | 2020-03-10 | 北京京东尚科信息技术有限公司 | Supermarket replenishment method and system |
| CN112464952A (en) * | 2020-11-25 | 2021-03-09 | 创新奇智(南京)科技有限公司 | Image alignment method and image matching identification method |
| CN113674049A (en) * | 2021-06-24 | 2021-11-19 | 杭州碑石数据技术有限公司 | Commodity shelf position identification method and system based on picture search and storage medium |
| CN114511820A (en) * | 2022-04-14 | 2022-05-17 | 美宜佳控股有限公司 | Goods shelf commodity detection method and device, computer equipment and storage medium |
| CN114494892A (en) * | 2022-04-15 | 2022-05-13 | 广州市玄武无线科技股份有限公司 | Goods shelf commodity display information identification method, device, equipment and storage medium |
| CN115249356A (en) * | 2022-09-21 | 2022-10-28 | 浙江莲荷科技有限公司 | Identification method, device, equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2805760C1 (en) * | 2023-02-13 | 2023-10-23 | Самсунг Электроникс Ко., Лтд. | Method and device for retail goods identification |
| CN117196779A (en) * | 2023-09-16 | 2023-12-08 | 广州小明网络科技有限公司 | Novel retail management system based on store |
| CN117196779B (en) * | 2023-09-16 | 2024-04-16 | 广州小明网络科技有限公司 | Novel retail management system based on store |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107292248B (en) | Commodity management method and system based on image recognition technology | |
| EP3910608B1 (en) | Article identification method and system, and electronic device | |
| CN109241374B (en) | Book information base updating method and library book positioning method | |
| CN111061890B (en) | Method for verifying labeling information, method and device for determining category | |
| US8941645B2 (en) | Comparing virtual and real images in a shopping experience | |
| CN108416902B (en) | Real-time object identification method and device based on difference identification | |
| US20230245443A1 (en) | Reducing scale estimate errors in shelf images | |
| JP2019055828A (en) | Shelf information estimation device and information processing program | |
| US20150046299A1 (en) | Inventory Assessment with Mobile Devices | |
| US8805123B2 (en) | System and method for video recognition based on visual image matching | |
| KR102380255B1 (en) | System and method for monitoring shelf goods inventory based on image | |
| CN103617420A (en) | Commodity fast recognition method and system based on image feature matching | |
| JP2013109773A (en) | Feature matching method and article recognition system | |
| CN115393007A (en) | Industry classification model training method and device | |
| CN115661624A (en) | Digital method and device for goods shelf and electronic equipment | |
| CN111428743B (en) | Commodity identification method, commodity processing device and electronic equipment | |
| CN110705620A (en) | Display image detection method and device based on copying recognition and storage medium | |
| CN113674049A (en) | Commodity shelf position identification method and system based on picture search and storage medium | |
| CN110378299B (en) | Clothing identification system under indoor lighting condition | |
| CN114360057A (en) | Data processing method and related device | |
| CN114494921A (en) | Goods shelf commodity identification method and system | |
| US20230274410A1 (en) | Retail shelf image processing and inventory tracking system | |
| US20230274227A1 (en) | Retail shelf image processing and inventory tracking system | |
| CN111126110A (en) | Commodity information identification method, settlement method and device and unmanned retail system | |
| US20230274226A1 (en) | Retail shelf image processing and inventory tracking system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |