CN117523273A

CN117523273A - Method and device for determining spatial position of article and electronic equipment

Info

Publication number: CN117523273A
Application number: CN202311459930.9A
Authority: CN
Inventors: 刘西洋; 李鹏; 廖宝鑫; 倪鼎; 王炎
Original assignee: Zhejiang Shenxiang Intelligent Technology Co ltd
Current assignee: Zhejiang Shenxiang Intelligent Technology Co ltd
Priority date: 2023-11-02
Filing date: 2023-11-02
Publication date: 2024-02-06

Abstract

In the method, an object identifier with a specific identifier ID is driven to send a visual identification signal, a target space position identified by the object identifier sending the visual identification signal is obtained through image identification, and a corresponding relation between the target space position and a target object is obtained by combining a pre-stored corresponding relation between the identifier ID and the object; by the method, the target object can be determined in a mode of identifying the spatial position of the object, and the accuracy of object identification is effectively improved. On the basis, the embodiment of the application also provides a method, a system, electronic equipment and a computer readable storage medium for prompting the article non-checkout information. When the user takes the target object in the video monitoring range, the specific information of the object taken by the user can be determined according to the spatial position of the object taken by the user, so that whether the user checks out correctly is further judged.

Description

Method and device for determining spatial position of article and electronic equipment

Technical Field

The application relates to the technical field of computers, in particular to a method for determining the spatial position of an article. The application also relates to an apparatus, an electronic device and a computer readable storage medium for determining the spatial position of an item. The application also relates to a method, a system, an electronic device and a computer readable storage medium for prompting the article non-checkout information.

Background

With the development of artificial intelligence technology, the technology of identifying articles by images has very wide application, such as shopping monitoring in shops, damage prevention monitoring of articles, and the like.

In the prior art, the type information of the object is determined by identifying the appearance characteristics of the object in the video, so that the object on site is monitored.

However, the above-described method for identifying an article by its appearance has a problem of low identification accuracy. For example, for fresh articles in a market environment, different fresh grades have very large price differences, but very fine appearance differences, and obviously, the fresh articles cannot be accurately distinguished by relying on an image recognition technology, so that the change condition of the on-site articles is very difficult to control by adopting a video technology.

Therefore, how to accurately identify the article type in a specific scenario is a problem to be solved.

Disclosure of Invention

The embodiment of the application provides a method for determining the spatial position of an article so as to improve the accuracy of identifying the type of the article in a specified scene. The embodiment of the application also provides a device for determining the spatial position of the article, electronic equipment and a computer readable storage medium. The embodiment of the application also provides a method and a system for prompting the article non-checkout information, electronic equipment and a computer readable storage medium.

The embodiment of the application provides a method for determining the spatial position of an article, which comprises the following steps: controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; and according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position.

Optionally, the controlling the object identifier with the selected identifier ID within the video monitoring range to send a visual identification signal includes: transmitting an opening instruction for opening a visual identification signal to object identifiers in a video monitoring range, wherein the opening instruction comprises a selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction.

Optionally, in the video monitoring picture, the obtaining, according to the visual identification signal, the target space position identified by the object identifier sending the visual identification signal includes: providing the video monitoring picture for a trained image recognition model, and recognizing a target object identifier which sends a visual recognition signal in the video monitoring picture; determining the placement position of the marked object associated with the target object identifier according to the position of the target object identifier in the video monitoring picture and the position offset relation between the spatial position of the object identifier and the spatial position of the target object; and taking the placement position as a target space position identified by the object identifier for sending the visual identification signal.

Optionally, the identifying the target object identifier in the video monitoring picture, which sends the visual identification signal, includes: identifying an item identifier contained in the video monitoring picture; generating a corresponding partial graph of the object identifier according to each identified object identifier; in the article identifier partial map, an article identifier that transmits a visual identification signal is identified as the target article identifier.

Optionally, the image recognition model comprises a first recognition model and a second recognition model; the first identification model is used for identifying object identifiers contained in the video monitoring picture; the second identification model is used for identifying the object identifier sending the visual identification signal in the object identifier partial graph.

Optionally, the second recognition model is obtained by training the selected initial visual recognition signal detection model by the following method: obtaining a data sample set for training the initial visual identification signal detection model, wherein the data samples in the data sample set comprise marked object identifier pictures, and signal transmitting points for transmitting visual identification signals are marked by using a first target frame; and providing the data sample set for the initial visual recognition signal detection model to train so as to enable the data sample set to meet the training requirement of recognizing the signal emission points in the object identifier local graph, thereby obtaining the second recognition model.

Optionally, the providing the data sample set to the initial visual recognition signal detection model for training to achieve training requirements for recognizing signal emission points in the object identifier local graph includes: providing the marked article identifier picture for the initial visual identification signal detection model to obtain an identification result output by the initial visual identification signal detection model, wherein the identification result comprises a first prediction frame aiming at the position of a signal emission point of the marked article identifier picture; obtaining a first loss value according to the difference between the first prediction frame in the identification result and the first target frame in the marked article identifier picture; adjusting parameters of the initial visual recognition signal detection model according to the first loss value; repeating the steps until the first loss value reaches the requirement of a preset threshold value, and obtaining a second recognition model after training is completed.

Optionally, labeling the first target frame on the article identifier picture sending the visual identification signal by adopting the following method: acquiring an article identifier picture for transmitting a visual identification signal; performing binarization processing on the object identifier picture to obtain a binarized picture; denoising other information except the signal emission point in the binarized picture to obtain a denoised picture containing the signal emission point; and adopting a marking frame with a preset shape to frame the signal emission point in the denoising picture to obtain the first target frame.

The embodiment of the application also provides a device for determining the spatial position of an article, which comprises: a control unit, an article space position association unit; the control unit is used for controlling the object identifiers with the selected identifier ID in the video monitoring range to send visual identification signals; the article space position association unit is used for acquiring a target space position identified by the article identifier for transmitting the visual identification signal according to the visual identification signal in a video monitoring picture; and associating the target object corresponding to the selected identifier ID to the target space position according to the pre-stored correspondence between the identifier ID and the target object.

Optionally, the control unit is specifically configured to send an opening instruction for opening the visual identification signal to the object identifier in the video monitoring range, where the opening instruction includes the selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction.

The embodiment of the application also provides a method for prompting the article non-checkout information, which comprises the following steps: controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position; if the target user is detected to be located in the video monitoring range in the video monitoring picture, if an intersection point exists between the position of the human body key point of the target user and the object detection frame of the target object and the object detection frame moves from the target space position to an area outside the video monitoring range, adding the information of the target object into an object taking list of the target user; the information of the target object is determined according to the target object associated with the target space position; judging whether the object in the object taking list has checkout record information or not, if not, judging that the object user has non-checkout risk aiming at the object, and sending risk prompt information.

Optionally, the determining whether the target object in the object taking list has checkout record information, if not, determining that the target user has an unclean risk for the target object includes: judging whether the objects in the object taking list have checkout record information or not when the target user is in the target area in the video monitoring picture, and if not, determining that the user has non-checkout risk for the objects in the object taking list; the target area is an area where the user performs settlement of the article or an area where the user has completed settlement of the article.

The embodiment of the application also provides an article non-checkout information prompting system, which comprises: the device comprises an article space position identification unit, an article taking identification unit and an article non-settlement risk identification unit; the article space position identification unit is used for controlling the article identifiers with the selected identifier ID in the video monitoring range to send visual identification signals; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position; the article taking and identifying unit is used for monitoring that a target user is located in the video monitoring range in a video monitoring picture, and adding information of the target article into an article taking list of the target user if an intersection point exists between the position of a human body key point of the target user and an article detecting frame of the target article and the article detecting frame moves from the target space position to an area outside the video monitoring range; the information of the target object is determined according to the target object associated with the target space position; the article non-settlement risk identification unit is used for judging whether settlement record information exists in the target article in the article taking list, if not, judging that the target user has non-settlement risk aiming at the target article, and sending risk prompt information.

Optionally, the article non-checkout risk identifying unit is specifically configured to determine, when it is detected in the video monitoring screen that the target user is in a target area, whether there is checkout record information on an article in the article taking list, and if there is no checkout record information, determine that the user has a non-checkout risk for the article in the article taking list; the target area is an area where the user performs settlement of the article or an area where the user has completed settlement of the article.

The embodiment of the application also provides electronic equipment, which comprises: a processor and a memory; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described above.

Embodiments of the present application also provide a computer-readable storage medium having stored thereon one or more computer instructions, wherein the instructions are executed by a processor to implement the above-described method.

Compared with the prior art, the embodiment of the application has the following advantages:

the method for determining the spatial position of the article provided by the embodiment of the application comprises the following steps: controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; and according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position.

In the method, the video monitoring range comprises a plurality of article identifiers, the ID of the article identifier which transmits the visual identification signal is selected, and the article identifier with the ID of the article identifier is controlled to transmit the visual identification signal. The visual identification signal sent by the article identifier can be obtained in the video monitoring picture, and the visual identification signal can determine the target space position identified by the article identifier, wherein the target space position is the target space position where the target article identified by the article identifier is located. The corresponding relation between the identifier ID and the target object is stored in advance, and the information of the target object corresponding to the object identifier with the selected identifier ID is obtained according to the corresponding relation. Thus, the target item is associated to the target spatial location. In the above process, the association relationship between the target space position and the target object enables the identification of the object class stored in each space position to be more accurate through the corresponding relationship between the identifier ID of the object identifier and the object class stored in advance.

The embodiment of the application provides an article non-checkout information prompting method, which comprises the following steps: controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position; if the target user is detected to be located in the video monitoring range in the video monitoring picture, if an intersection point exists between the position of the human body key point of the target user and the object detection frame of the target object and the object detection frame moves from the target space position to an area outside the video monitoring range, adding the information of the target object into an object taking list of the target user; the information of the target object is determined according to the target object associated with the target space position; judging whether the object in the object taking list has checkout record information or not, if not, judging that the object user has non-checkout risk aiming at the object, and sending risk prompt information.

In the method, firstly, an object identifier with a specific identifier ID is driven to send a visual identification signal, a target space position identified by the object identifier for sending the visual identification signal is obtained through image identification, and the corresponding relation between the target space position and the target object is obtained by combining the corresponding relation between the prestored identifier ID and the object; by the method, the object article can be determined in a mode of identifying the spatial position of the article, and the accuracy of article identification is effectively improved. On the basis, when a user takes a target object in the video monitoring range, specific information of the object taken by the user can be determined according to the spatial position of the object taken by the user in the video monitoring picture; thereby further determining whether the user is checkout for the picked-up target item. The method provided by the application can effectively improve the accuracy of the risk judgment of the article checkout, and particularly can effectively avoid misjudging the type of the article taken by the user.

Drawings

Fig. 1 is an application scenario schematic diagram of a method for determining a spatial position of an article according to an embodiment of the present application.

Fig. 2 is a schematic diagram of a method for intelligently monitoring a commercial superliving fresh goods shelf according to an embodiment of the present application.

Fig. 3 is a schematic view of a scene for identifying a position of an item identifier and a target spatial position of a target item in a video monitoring screen according to an embodiment of the present application.

Fig. 4 is a schematic diagram of labeling a first target frame with an article identifier picture of a visual identification signal according to an embodiment of the present application.

Fig. 5 is a schematic diagram of a scenario for updating a customer pick up list according to an embodiment of the present application.

Fig. 6 is a schematic diagram of functional units corresponding to the unsettled risk identification module 203 in fig. 2.

Fig. 7 is a flowchart of a method for determining a spatial position of an item provided in a first embodiment of the present application.

Fig. 8 is a schematic view of an apparatus for determining a spatial position of an article according to a second embodiment of the present application.

Fig. 9 is a flowchart of a method for prompting article non-checkout information according to a third embodiment of the present application.

Fig. 10 is a logic frame diagram of an article non-checkout information prompting system according to a fourth embodiment of the present application.

Fig. 11 is a schematic diagram of an electronic device according to a fifth embodiment of the present application.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is, however, susceptible of embodiment in many other ways than those herein described and similar generalizations can be made by those skilled in the art without departing from the spirit of the application and the application is therefore not limited to the specific embodiments disclosed below.

First, some technical terms related to the present application will be explained:

article identifier: an article for calibrating identification information to an article, such as an article displayed in a mall. Each item identifier has its identifier ID identifying itself to distinguish it from any other item identifier; the article identifier is used as an identification of a particular article, as desired. In the embodiment of the application, a typical example of the article identifier is an electronic price tag, the electronic price tag is provided with a display part for providing a position for displaying information, and a user can know the information of the article identified by the electronic price tag according to the information provided by the display part. Further, in order to achieve the object of the embodiments of the present application, the electronic price tag can receive a wireless signal through a short-range wireless communication method, and further has an indicator lamp (an LED lamp may be used) that can be lighted under the control of the wireless signal. In the embodiment of the application, each article identifier arranged on the site can be controlled by an external instruction to light the indicator lamp. In addition, in the application, a special database is arranged for recording the corresponding relation between each article identifier and the specific article identified by the article identifier; specifically, the object identifier is embodied to a certain target object identifier by using the identifier ID, and the specific object identified by the object identifier can be recorded by using the SKU of the object, that is, the corresponding relationship between the identifier ID and the target object is recorded in a special database; for an item manager using an item identifier (e.g., a tally clerk using an item identifier), the item identifier and the corresponding target item should be placed in spatial correspondence so that a user located on site can be assured that the item identified by the item identifier is accurately determined by the item identifier and is consistent with the records of the database. Under different scenes, the corresponding relation between a specific target object and an object identifier can be divided into different levels, in general, under the commodity sales scenes of a market and the like, the target object can be identified by the SKU of the commodity, the SKU is the minimum classification unit of the commodity, the commodities belonging to the same SKU are not different from each other and have the same meaning for a user, and at the moment, the corresponding relation between the object identifier and the target object is the corresponding relation between the identifier ID of the object identifier and the commodity SKU; of course, in the case of a large commodity such as an automobile, the target object may be defined on a more specific classification level of a certain automobile or the like.

Target spatial position: refers to a specific placement location corresponding to a specific item (e.g., a commodity of a SKU), and in the embodiment of the present application, each target spatial location is indicated by an item identifier, for placement of a specific target item.

In the embodiment of the application, a shopping scene of a market is taken as a typical scene, especially a fresh shopping scene, the provided target objects specifically refer to object SKUs, the object SKUs (Stock Keeping Unit, stock units), wherein the stock units are basic units for the entry and exit of the stock, the objects identified by the same SKU can be considered to be not different from each other (for commercial selling), for example, a medium-grade first-grade spareribs are one SKU, and for example, an S-code white short sleeve of a certain brand is one SKU.

And (5) identifying the picking behavior: and identifying whether the user has the action of taking the article or not, and recording the article information taken by the user.

Article monitoring and tracking: and detecting the motion trail of the articles on the goods shelf in real time.

Taking a goods list: a list of items that the user has picked up to pay.

Checkout record list: the user has paid for a list of items.

In order to facilitate understanding of the methods provided by the embodiments of the present application, the background of the embodiments of the present application is described before the embodiments of the present application are described.

With the development of artificial intelligence technology, the technology of identifying articles by images has very wide application, such as shopping monitoring articles in shops, damage-prevention monitoring of articles, and the like. In the prior art, the type information of the object is determined by identifying the appearance characteristics of the object in the video, so that the object on site is monitored. However, in the above prior art, in the case that the difference of the appearance characteristics of the commodities of different SKUs such as fresh sales is often small, there is a problem that the recognition accuracy is low. Therefore, the embodiment of the application provides a technical scheme capable of identifying the on-site object more accurately.

The following describes in detail an application scenario of the method for determining a spatial position of an article of the present application. The method for determining the spatial position of the object can be applied to the field of identifying the spatial position information of the object in the internet of things, for example, the field of identifying the spatial position information of the object in the goods shelf, or other related technical fields requiring positioning detection of the position of the object.

In the following, first, a specific application scenario of the method for determining a spatial position of an article provided in the embodiment of the present application is illustrated.

Fig. 1 is an application scenario schematic diagram of a method for determining a spatial position of an article according to an embodiment of the present application. As shown in fig. 1, in the present application scenario, a first item identifier 201, a second item identifier 202, and a third item identifier 203 are provided on a first shelf 300. Accordingly, first item identifier 201 corresponds to first item 101, second item identifier 202 corresponds to second item 102, and third item identifier 203 corresponds to third item 103. Wherein the item identifiers may be electronic price tags or electronic bottle labels, each having its own unique ID, referred to as identifier ID, and which are capable of receiving close range wireless signals, in embodiments of the present application, which are also capable of issuing visual identification signals to be determined in performing image recognition; typically, a small LED lamp may be provided as an element for emitting a visual identification signal.

In the embodiment of the application, assuming that the difference of appearance features between the first article, the second article and the third article is smaller, when a specific article cannot be distinguished through the appearance features, in order to improve the identification accuracy of identifying the spatial position of the article, thereby determining SKU information of the article taken by a user according to the spatial position information of the article acquired by the user in the video monitoring picture.

As shown in fig. 1, the first article identifier comprises a signal emission point 201-1 emitting a visual identification signal, and likewise, the second article identifier comprises a signal emission point 202-1 emitting a visual identification signal, and the third article identifier comprises a signal emission point 203-1 emitting a visual identification signal. The signal emitting point may be an LED lamp, and the article identifier has a power supply and a control circuit therein, and is capable of receiving a wireless signal, and controlling the signal emitting point of the article identifier having the identifier ID to emit a visual identification signal according to an on visual identification instruction signal in the wireless signal and according to a selected identifier ID included in the on instruction; thus, the spatial position of the object identifier can be accurately identified according to the visual identification signal in the video monitoring picture.

For example, the item identifier is an electronic price tag and the signal emitting point is a flashing light. The flashing lamps which can realize the flashing function are respectively arranged on the first electronic price tag, the second electronic price tag and the third electronic price tag, for example, the flashing lamp of the first electronic price tag, the flashing lamp of the second electronic price tag and the flashing lamp of the third electronic price tag. And the flashing state of the on/off state is presented by the flashing lamp of the electronic price tag, and the position of a specific electronic price tag in the goods shelf is determined.

For another example, the article identifier is an article sticker carrying ID information, and the article sticker may receive a short-range wireless signal and transmit a visual identification signal based on the received short-range wireless signal.

The control unit records the correspondence between the identifier ID of the article identifier and the target article (specifically, for example, commodity SKU) through a database before sending a control instruction to the article identifier, and controls the control unit to send out the visual identification signal according to the identifier ID. The relation between the identifier ID of the article identifier and the position of the article identifier in the visual monitoring picture is preset, and the specific setting process is as follows:

in the placement area (such as a shelf) of the article, article identifiers are placed in place, each article identifier indicates an article space position, and a determinable position offset relationship exists between the space position of the article identifier and the space position of the associated target article, so that the target space position of the identified target article can be obtained according to the space position of the article identifier. In the identification preparation stage, the control unit sends out a visual identification signal according to each article identifier ID by sending a short-distance wireless signal with article identifier ID information, and in a monitoring picture, according to the visual identification signal, the spatial position of the article identifier sending the visual identification signal is obtained through an identification model, namely the spatial position of the article identifier with the identifier ID, and further, the target spatial position of the article identified by the article identifier can be determined through a position offset relation.

For example, before the shelf is used normally, the first shelf is placed with a plurality of electronic price tags and corresponding articles in sequence. Then, the blinking lamps of the electronic price tags of the first shelf are sequentially controlled to blink in a mode of sending signals through the base station, so that the corresponding relation between the electronic price tags (the electronic price tags contain IDs (namely identifier IDs) and the spatial positions of the articles marked by the electronic price tags is determined in an image recognition mode.

According to the method for determining the spatial position of the article, under the condition that the appearance characteristics of the article cannot be clearly identified, the accuracy of identifying the article is improved through the relation between the spatial position of the article and the article. On the basis, whether the user takes the article and whether the user pays the article or not is judged according to the spatial position of the article and the intersection point between the user and the article, and if the user pays the article, an unpaid risk reminder can be sent.

The embodiment of the application is particularly suitable for scenes in which the appearance characteristics of the article cannot be clearly identified. For example, for fresh foods, the same fresh food has different pricing and different commodity SKUs due to different quality and freshness, so that the fresh food cannot be distinguished by appearance characteristics of the food, and the monitoring effect of the video on the selling condition of the food is affected; after the embodiment of the application is adopted, the problems can be effectively solved.

Fig. 2 is a schematic diagram of a method for intelligently monitoring a commercial superliving fresh goods shelf according to an embodiment of the present application. Fig. 2 includes three components, where 201 in fig. 2 is an electronic price tag and merchandise identification module, 202 in fig. 2 is a pick identification module, and 203 in fig. 2 is an outstanding risk identification module.

Wherein the electronic price tag and commodity identification module 201 performs the following functions in order: the electronic price tag is arranged in space with the commodity; electronic price tag flash identification; the electronic price tag is matched with the commodity in a related way; outputting the commodity position and the SKU.

The electronic price tag and commodity identification module 201 is configured to determine the target spatial position of the electronic price tag and the target commodity identified by the electronic price tag, so that in a subsequent step, detailed information of the target commodity taken by the customer can be determined according to the target spatial position of the target commodity taken by the customer in the video monitoring screen. The detailed information of the target item herein mainly includes SKU information of the target item.

The electronic price tag is associated with and matched with the commodity by the core function executed by the commodity identification module 201, so as to obtain the space position information of each specific commodity, or obtain the information of what commodity (expressed by SKU) is placed in each space position.

The method comprises the following steps: and controlling the flashing points of the electronic price tags with the selected electronic price tag ID in the video monitoring range to send flashing signals which are visual identification signals and show the flashing state. The electronic price tag can acquire a flicker signal sent by the electronic price tag in a video monitoring picture, and the target space position marked by the electronic price tag can be determined according to the flicker signal, wherein the target space position is the target space position of the target commodity marked by the electronic price tag. The corresponding relation between the electronic price tag ID and the target commodity is stored in advance, and the information of the target commodity corresponding to the electronic price tag with the selected electronic price tag ID can be obtained according to the corresponding relation. Thus, the above steps can correlate the target commodity to the target spatial location. According to the process, through the flashing signal sent by the electronic price tag, the target space position marked by a specific electronic price tag can be accurately obtained, and the association relation between the target space position and the target commodity can be established by combining the target object corresponding to the prestored electronic price tag ID. Through the scheme, the commodity fetched by the customer can be determined through spatial position identification, and the identification accuracy of the target commodity is remarkably improved.

The pick-and-recognize module 202 includes the following functions: detecting and tracking commodities; detecting key points of a human body; judging interaction between a human hand and the commodity; and counting the inventory of the goods and the goods.

The pickup identification module 202 is configured to monitor whether an intersection exists between the customer and the target commodity according to the video monitoring screen, and determine whether the user picks up the target commodity according to the movement track of the target commodity. The detailed description thereof may refer to the description of fig. 5.

Please refer to fig. 5, which is a schematic diagram of a scenario for updating a customer pick up list according to an embodiment of the present application.

A schematic diagram of the functional units implementing this function is shown in fig. 5. Comprises the following functional units:

s501: commodity SKU information is identified. S502: and (5) commodity detection and tracking. S503: user human body detection and tracking. S504: extracting key points of human bones. S505: and judging the intersection point of the key points of the human hands and the commodity frame. S506: the two are in the intersection point judgment and take goods. S507: and (5) judging that the commodity track disappears and the picking is completed. S508: updating the customer pick list.

In fig. 5, SKU information of a target commodity is implemented according to a touch of a user on the target commodity in a video monitoring screen, spatial position information of the target commodity, and association relationship information between the spatial position information of the target commodity and the target commodity. Then, the moving track of the target commodity is detected and tracked.

Meanwhile, human body key points of the user are detected and tracked, specifically human body skeleton key points of the user are extracted, and when the intersection point between the user and the target commodity is detected, the intersection point between the hand key points of the user and the commodity detection frame of the target commodity is detected and judged.

If there is an intersection between the location of the user' S human body key point and the item detection frame of the target item (S505), it is determined that the user is in the process of picking up the target item (S506). If the object detection frame of the target object moves from the target spatial position to an area outside the video detection range, in other words, the moving track of the target object disappears in the video monitoring screen, it is determined that the user has picked up the target object (S507), and pick up list information of the user is updated (S508).

The unsettled risk identification module 203 includes: tracking the full-field track of the customer; occurs at the exit/checkout area; comparing the goods with the checkout bill; and (5) early warning of risk of not settling.

The non-checkout risk recognition module 203 is configured to determine whether the customer checks out the target commodity, and if not, send a non-checkout prompt message, where the prompt message may be sent to a staff member, a user, etc. The specific judging process is shown as 203 in fig. 2, and the full-field track of the customer is tracked; (judgment target client) is present in the export/checkout area; comparing the pick (of the target customer) with the checkout bill; and (5) carrying out early warning of the risk of not settling according to the comparison result.

For a more detailed process, reference may be made to the description of fig. 6.

Please refer to fig. 6, which is a schematic diagram of functional units corresponding to the unsettled risk identification module 203 in fig. 2. In fig. 6, it includes: s601: tracking the full-field track of the customer in real time; s602: customers are present at the exit and do not pass through the checkout area; s603: whether a pick bill and a transaction record exist or not; s604: risk of not settling accounts at all; s605: sending risk early warning; s606: the customer is present in the checkout area; s607: comparing the pick-up and checkout list after the transaction is finished; s608: part of the risk of not settling.

S602 to S604 in fig. 6 are processes for determining whether a user checks out a target commodity when the user is in a first area, which may refer to an area where various forms of user should have completed settlement of the commodity, typically, a mall exit.

And judging whether the target object in the object taking list has checkout record information, if not, judging that the target user has non-checkout risk aiming at the target object, wherein the method comprises the following steps:

tracking the full-field track of the customer in real time (S601), so as to acquire the mobile position information of the target user in the video monitoring picture; and monitoring that the target user is in the first area and the moving track of the target user meets a certain condition (such as 'appearing at an exit and not passing through a checkout area' in S602 in fig. 6) in the video monitoring picture, judging whether the target user takes a bill and a transaction record (S603), and if so, determining that the target user has a risk of not being checked out at all for the articles in the article taking bill (S604).

S606 to S608 in fig. 6 are processes for determining whether the target user is checking out the target commodity when the target user is in the second area, which is an area where the target user performs the payment process, for example, a check-out area of a mall.

The determining whether the target object in the object taking list has checkout record information, if not, determining that the target user has partial non-checkout risk for the target object includes:

tracking the full-field track of the customer in real time (S601), so as to acquire the mobile position information of the target user in the video monitoring picture; monitoring that the target user is in a second area in the video monitoring picture (S606), wherein the second area is an area where the target user performs settlement of the articles, such as a settlement area of a mall; when the target user checkout transaction is finished, the system compares the object taking list of the target user with the checkout list (S607), judges whether the corresponding checkout record information exists in the checkout record list for the target object in the object taking list, and determines that the target user has an unclosed risk for the target object if the situation exists that the target object in the object taking list does not exist in the checkout list (S608).

Through the two conditions, whether the target user checks out the target object is judged, and if the target object is not checked out, the risk early warning of the non-checking out is sent. According to the method provided by the embodiment of the application, the corresponding relation between the target space position marked by the object marker and the target object is determined through the visual identification signal in the video monitoring picture, so that the object taken by the user can be determined through the identification of the space position in the video monitoring picture, and the identification accuracy of determining the target object taken by the user is effectively improved. On the basis, the contact track of the user and the target object and the action track of the user are tracked in the video monitoring picture, so that the monitoring and tracking of the user and the target object are realized, and whether the user finishes checkout on the target object can be accurately determined.

First embodiment

Referring to fig. 7, a flowchart of a method for determining a spatial position of an object according to a first embodiment of the present application is provided, and the method provided by the present embodiment is described in detail below with reference to fig. 7. The embodiments referred to in the following description are intended to illustrate the method principles and not to limit the practical use. The method for determining the spatial position of the object provided by the first embodiment corresponds to the scene embodiment, and therefore, the relevant part can refer to the scene embodiment. The method for determining the spatial position of an article shown in fig. 7 includes steps S701 to S703.

Step S701, controlling an item identifier having a selected identifier ID within the video monitoring range to transmit a visual identification signal.

This step is used to control the article identifier to transmit a visual identification signal, and the identifier ID of the article identifier that needs to transmit the visual identification signal has been determined before the article identifier is controlled to transmit the visual identification signal.

Specifically, the article identifier can be controlled to send a visual identification signal in a short-distance wireless communication mode; the short-range wireless communication method includes, for example, wi-fi or bluetooth, and these signals may be transmitted by a host device (for example, a base station) in a video monitoring site, and the transmitted start command of the start visual identification signal includes the identifier ID of the specific specified article identifier.

Before this, the item identifier and its identified target item have been properly positioned at the video monitoring site, comprising: and placing the object identifiers according to preset positions in advance, and placing the target object indicated by the object identifiers at the spatial positions marked by the object identifiers.

In the process of establishing the relation between the target space position and the target object, under video monitoring, a control unit positioned at the host equipment sends an opening instruction for opening a visual identification signal in a short-distance wireless signal mode according to the identifier ID of each object identifier, and correspondingly, a video monitoring picture at the moment is obtained; the spatial position of the item identifier with the identifier ID in the video monitoring screen is determined by determining the item identifier of the transmitted visual identification signal in the video monitoring screen.

The method comprises the following steps of controlling an article identifier with a selected identifier ID in a video monitoring range to send a visual identification signal, wherein the visual identification signal is sent by the article identifier with the selected identifier ID in the video monitoring range, and the method can be realized by the following steps of:

transmitting an opening instruction for opening a visual identification signal to an object identifier in a video monitoring range through control equipment (such as a base station), wherein the opening instruction comprises a selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction. The specific mode of sending the start command generally adopts a short-range wireless communication mode, for example, a communication mode such as WiFi or bluetooth.

As shown in fig. 1, the video monitoring range includes a plurality of object identifiers, an object identifier which needs to send a visual identification signal is selected in a mode of selecting an identifier ID, an opening instruction sent in a mode of a short-distance wireless signal is sent to the object identifier, and the opening instruction includes the selected identifier ID; and after the article identifier receives the opening instruction, if the identifier ID in the article identifier is found to be consistent with the identifier ID of the article identifier after decoding, driving the vision element of the article identifier to execute the operation of sending the vision identification signal. The visual element can take various forms in the prior art, and typically can take an LED indicator or a light emitting diode; the visual recognition signal includes various forms such as a blinking signal exhibiting a blinking state, a lighting signal.

Step S702, in the video monitoring screen, according to the visual identification signal, acquiring the target spatial position identified by the object identifier sending the visual identification signal.

After controlling the article identifier having the selected identifier ID to emit a visual identification signal in step S701, the present step identifies the target spatial position identified by the article identifier emitting the visual identification signal in the video monitoring screen. Wherein the target spatial location is a spatial location of the target item identified by the item identifier.

In the video monitoring picture, according to the visual identification signal, the target space position identified by the object identifier for sending the visual identification signal is obtained by the following method:

providing the video monitoring picture for a trained image recognition model, and recognizing a target object identifier which sends a visual recognition signal in the video monitoring picture; determining the placement position of the marked object associated with the target object identifier according to the position of the target object identifier in the video monitoring picture and the position offset relation between the spatial position of the object identifier and the spatial position of the target object; and taking the placement position as a target space position identified by the object identifier for sending the visual identification signal.

And identifying the video monitoring picture by adopting a trained image identification model, and acquiring the position of an article identifier sending out a visual identification signal in the video monitoring picture and the target space position of a target article identified by the article identifier.

Fig. 3 is a schematic view of a scene for identifying a position of an item identifier and a target spatial position of a target item in a video monitoring screen according to an embodiment of the present application. Fig. 3 includes: s301: acquiring a video monitoring picture; s302: detecting a result in the first stage; s303: acquiring an article identifier graph; s304: detecting a result in the second stage; s305: and detecting the result in the third stage to obtain the spatial position of the target object.

In fig. 3, the process of identifying the video monitoring picture by the trained image identification model includes three stages, namely, a first stage, obtaining information of all the object identifiers in the video monitoring picture, generating an object identifier local graph, a second stage, obtaining positions of the object identifiers sending visual identification signals in the object identifier local graph, and a third stage, obtaining target space positions of target objects identified by the object identifiers.

The final purpose of the first two stages is to obtain the position of the object identifier in the object identifier partial graph, which emits a visual identification signal, and the third stage obtains the target spatial position of the target object.

The identification of the target object identifier for sending the visual identification signal in the video monitoring picture can be realized by the following modes:

identifying an item identifier contained in the video monitoring picture; generating a corresponding partial graph of the object identifier according to each identified object identifier; in the article identifier partial map, an article identifier that transmits a visual identification signal is identified as the target article identifier.

In the process of identifying the target object identifier of the visual identification signal sent in the video monitoring picture, the identification can be completed through a trained image identification model; the image recognition model can be divided into a first recognition model and a second recognition model according to a specific implementation process; the first identification model is used for identifying object identifiers contained in the video monitoring picture; the second identification model is used for identifying the object identifier sending the visual identification signal in the object identifier partial graph.

With continued reference to fig. 3, the following describes the complete recognition process of the video monitoring frames by the trained image recognition model.

S301: acquiring a video monitoring picture; the video monitoring pictures comprise pictures obtained in the process of controlling the object identifiers with the selected identifier ID in the video monitoring range to send visual identification signals.

S302: and acquiring a first stage detection result by using the first identification model, and framing all object identifiers in the video monitoring picture by using a detection frame.

The first recognition model may employ various object recognition models (or image classification models) in the prior art; these models are typically trained by deep learning methods and can be used to automatically identify and classify different classes or objects in the image. Common image classification models include convolutional neural networks (Convolutional Neural Networks, CNN for short) and their variants, such as ResNet, VGGNet, inception, etc. These image classification models can accurately classify different kinds of things by learning the way in which important features are extracted from the original image data. They are often subject to extensive training sets and optimizations to improve classification accuracy and have the ability to perform well on a variety of image classification tasks.

In general, a machine learning model for identifying a certain kind of things in an image is called an image classification model or an object identification model. These models are typically trained by deep learning methods and can be used to automatically identify and classify different classes or objects in the image. Common image classification models include convolutional neural networks (Convolutional Neural Networks, CNN for short) and their variants, such as ResNet, VGGNet, inception, etc. It should be noted that the specific algorithm and model names may change over time and development, as new algorithms and models continue to emerge in the machine learning field. Thus, as technology evolves and research progresses, there may be other specific nomenclature that is used for a specific task or innovation model.

In particular, in this embodiment, an image sample marked with an object identifier in an image may be used as a training sample, and various machine models that may be used for object recognition in the prior art are used as initial models, and training is performed to obtain a first recognition model required in this application.

S303: generating an article identifier partial graph of all the article identifiers.

After the detection frame is selected by adopting the first identification model for the object identifier in the monitoring image, the part of the video monitoring image positioned in the detection frame can be extracted, so that a corresponding object identifier partial image is generated.

S304: and aiming at the object identifier partial graph, a second recognition model is adopted to obtain a second stage detection result, and specifically, the object identifier sending the visual recognition signal is recognized.

Since the article identifier that transmitted the visual identification signal has a conspicuous visual identification signal, it can be distinguished from other article identifiers by the visual identification signal, thereby determining the article identifier as having the selected identifier ID.

In some cases, after the visual identification signal possibly sent out in the partial graph of the article identifier is identified, the visual identification signal is used as a candidate article identifier, and further, the following judgment can be performed on the candidate article identifier: and judging whether the characteristics of the visual identification signal sent by the object identifier sending the visual identification signal meet the preset condition, and if so, determining the position of the candidate object identifier as the position of the object identifier sending the visual identification signal in the video monitoring picture.

The preset conditions may be selected as needed, for example, the preset conditions include at least one of the following conditions: the signal transmitting point of the article identifier transmits a visual identification signal for a preset number of times within a preset time length; the time length of the signal sent by the signal emitting point of the article identifier is longer than the preset time length.

For example, the signal emitting point is a blinking lamp, and the visual recognition signal is a blinking signal exhibiting a blinking state. The flashing lamp emits a flashing signal for a preset number of times within a preset time period. For another example, the signal emitting point has the functions of lighting and turning off, the signal emitting point sends a signal with a lighting time length longer than a preset time length, and the signal sent by the signal emitting point of the article identifier is a signal meeting a preset condition.

S305: and obtaining a detection result of the third stage, and obtaining the spatial position of the target object.

After determining the position of an article identifier of a selected identifier ID, determining the placement position of an identified article associated with a target article identifier according to the position information of the article identifier and the position offset relationship between the position of the article identifier and the spatial position of the target article stored in advance; further, the placement position is used as a target space position identified by the object identifier for sending the visual identification signal.

The placement position of the marked object associated with the target object identifier is obtained according to the position information of the object identifier, and various modes can be adopted; generally, the object identifier and the identified object have a spatial association relationship, and the spatial relationship between the object identifier and the identified object can be called a position offset relationship, so that the spatial position of the target object can be obtained from the object identifier through a certain coordinate offset calculation; the coordinate offset calculation can be directly realized in a mathematical calculation mode, or can also be realized in a mathematical calculation and model-assisted mode, for example, the object identifier is an electronic price tag, the spatial position of the target object is the shelf position of the shelf, the shelf position in the monitoring picture can be identified through a trained object identification model, and then according to the position offset relation, which shelf position corresponds to the target electronic price tag is determined, so that the corresponding target spatial position can be obtained through the object identifier.

The process described in fig. 3 is a process in which the trained image recognition model obtains the target spatial position of the target object according to the video monitoring picture. The second recognition model is obtained by training a selected initial visual recognition signal detection model by the following method:

Obtaining a data sample set for training an initial visual identification signal detection model, wherein the data samples in the data sample set comprise marked article identification pictures, and signal transmitting points for transmitting visual identification signals are marked by using a first target frame; and providing the data sample set for the initial visual recognition signal detection model to train so as to enable the data sample set to meet the training requirement of recognizing the signal emission points in the object identifier local graph, thereby obtaining the second recognition model.

The data samples in the data sample set are obtained after labeling the target frames of the object identifiers which send out the visual identification signals and are contained in the video monitoring picture. The training purpose of the second recognition model is to be able to recognize the object identifier that emits the visual recognition signal in the video monitoring picture, and this recognition process obviously needs to have a sufficient recognition effect on the visual recognition signal, and then recognizes the recognized object identifier that emits the visual recognition signal as the target object identifier.

Therefore, in the data sample set obtained here, the target frame for labeling the sample data is specifically a target frame for the position of the signal emission point for transmitting the visual identification signal, which is referred to as a first target frame in this embodiment.

The step of providing the data sample set to the initial visual recognition signal detection model for training to achieve the training requirement of recognizing the signal emission points in the object identifier local diagram comprises the following steps:

providing the marked article identifier picture for the initial visual identification signal detection model to obtain an identification result output by the initial visual identification signal detection model, wherein the identification result comprises a first prediction frame aiming at the position of a signal emission point of the marked article identifier picture; obtaining a first loss value according to the difference between the first prediction frame in the identification result and the first target frame in the marked article identifier picture; adjusting parameters of the initial visual recognition signal detection model according to the first loss value; repeating the steps until the first loss value reaches the preset threshold value requirement, and obtaining a second recognition model after training is completed.

In order to improve the training accuracy of the image recognition model, a large number of labeling samples need to be provided; in the embodiment of the application, the method for automatically labeling the candidate object identifier picture sending the visual identification signal is particularly used for training the data sample required by the second identification model, so that a large number of available data samples are obtained as soon as possible, and the training speed of the second identification model is improved. The method specifically comprises the following steps of labeling a first target frame on an article identifier picture which sends a visual identification signal. A schematic diagram of labeling a first target frame with an article identifier picture of a visual identification signal according to an embodiment of the present application is described with reference to fig. 4.

Acquiring an article identifier picture for transmitting a visual identification signal; performing binarization processing on the object identifier picture to obtain a binarized picture; denoising other information except the signal emission point in the binarized picture to obtain a denoised picture containing the signal emission point; and adopting a marking frame with a preset shape to frame the signal emission point in the denoising picture to obtain the first target frame.

In which Binarization (Binarization) is the conversion of an image into a binary image, i.e. the pixel values in the image are divided into two values, usually black and white, to highlight the edges and features of the target object. For the recognition of the position of the visual recognition signal (in this embodiment, the light signal actually) a fixed threshold method can be used for binarization, that is, a threshold value is selected, the pixels with the pixel values greater than the threshold value in the image are set to be white, and the pixels with the pixel values less than or equal to the threshold value are set to be black. Of course, other methods such as an adaptive threshold method may be employed. In the binarized image, there may be some noise points or intermittent pixels. To remove these disturbances, a denoising process may be performed; specific denoising methods are numerous, and typically, morphological operations are adopted: including Erosion (Erosion) and expansion (displacement) operations, connectivity and shape of binary images can be improved by changing the shape and size of structural elements; the erosion operation may eliminate small noise points and the dilation operation may fill voids in the image. The connected region analysis method can also be adopted, namely noise points and isolated pixels can be identified and removed by detecting the connected region in the binary image; this may be achieved by a pixel connectivity algorithm, such as a connectivity component labeling algorithm. Median filtering may also be employed: the median value in the pixel neighborhood is used for replacing the value of the current pixel, so that noise points can be effectively eliminated; the median filtering may be performed before or after binarization.

Of course, a suitable binarization and denoising method needs to be selected according to specific image characteristics and noise conditions. In addition, parameter adjustment and multiple iterations can be performed according to specific requirements, so that a better effect is obtained. In the present embodiment, the binarization process recommends the use of a fixed threshold method, and the denoising process recommends the use of a morphological method including etching and swelling operations.

Based on this, the trained second recognition model is combined with the first recognition model, the position of the article identifier transmitting the visual recognition signal can be obtained according to the visual recognition signal appearing in the video monitoring picture, the placement position of the identified article associated with the target article identifier is determined according to the position of the article identifier and the position offset relationship between the position of the article identifier and the spatial position of the target article, and the placement position is taken as the target spatial position identified by the article identifier transmitting the visual recognition signal,

step S703, associating the target object corresponding to the selected identifier ID to the target space position according to the pre-stored correspondence between the identifier ID and the target object.

The step is used for associating the target object to the target space position according to the target space position identified by the object identifier with the identifier ID and the target object corresponding to the identifier ID in advance.

Based on this, an association relationship is established between the target object and the target spatial position. By the mode, all articles in the video monitoring picture and the space positions corresponding to the articles can be built in a relation one by one; namely, the effect that the stored article can be known from the spatial position of the video monitoring picture can be achieved; for example, from a certain shelf location (target space location), the item stored therein can be known as a commodity of a certain SKU. In this way, in the subsequent user behavior tracking, the specific object related to the user behavior can be deduced according to the spatial position related to the user behavior identified in the video monitoring process.

For example, in a market scene, through a user's operation of taking a target object in a video monitoring screen, detailed information (the core is SKU information, and after SKU information is obtained, the rest of information can be obtained from a database) of the target object can be determined through a target spatial position of the target object in the video monitoring screen.

In the method, a plurality of article identifiers are included in the video monitoring range, the article identifier which transmits the visual identification signal is selected through the identifier ID, and the article identifier with the identifier ID is controlled to transmit the visual identification signal. And a visual identification signal sent by the article identifier can be obtained in the video monitoring picture, and the target space position identified by the article identifier can be determined through the visual identification signal, wherein the target space position is the target space position of the target article identified by the article identifier. The corresponding relation between the identifier ID and the target object is stored in advance, and the information of the target object corresponding to the object identifier with the selected identifier ID is obtained according to the corresponding relation; by the method, the target object can be associated to the target space position. In the above process, the association relationship between the target space position and the target object is determined by the corresponding relationship between the identifier ID of the pre-stored object identifier and the object category, so that the object category stored in each space position is more accurately identified.

Second embodiment

Referring to fig. 8, a schematic diagram of an apparatus for determining a spatial position of an object according to a second embodiment of the present application is provided, and the apparatus provided in this embodiment is described in detail below with reference to fig. 8. The embodiments referred to in the following description are intended to illustrate the method principles and not to limit the practical use. The apparatus for determining the spatial position of an article provided in the second embodiment corresponds to the first embodiment and the scene embodiment, and thus, the relevant part can refer to the first embodiment and the scene embodiment. The apparatus 800 for determining a spatial position of an item shown in fig. 8 comprises a control unit 801, an item spatial position correlation unit 802.

The control unit 801 is configured to control an article identifier having a selected identifier ID within a video monitoring range to transmit a visual identification signal.

The control unit 801 is arranged to control the transmission of a visual identification signal by the item identifier, the control unit having determined the identifier ID of the item identifier for which a visual identification signal needs to be transmitted before a control instruction is transmitted to the item identifier. Specifically, the article identifier can be controlled to send a visual identification signal in a short-distance wireless communication mode; the short-range wireless communication method includes, for example, wi-fi or bluetooth, and these signals may be transmitted by a host device in a video monitoring site, and the transmitted start command for starting the visual identification signal includes the identifier ID of the specific specified article identifier.

In the process of establishing the relation between the target space position and the target object, in video monitoring, a control unit positioned in the host equipment sends an opening instruction for opening a visual identification signal in a short-distance wireless signal mode according to the identifier ID of each object identifier, and correspondingly, a video monitoring picture at the moment is obtained; the spatial position of the item identifier with the identifier ID in the video monitoring screen is determined by determining the item identifier of the transmitted visual identification signal in the video monitoring screen.

The control unit 801 controls the object identifier having the selected identifier ID in the video monitoring range to send a visual identification signal, specifically, sends an opening instruction for opening the visual identification signal to the object identifier in the video monitoring range, where the opening instruction includes the selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction. The specific mode of sending the start command generally adopts a short-range wireless communication mode, for example, a communication mode such as WiFi or bluetooth.

The article spatial position association unit 802 is configured to obtain, in a video monitoring screen, according to the visual identification signal, a target spatial position identified by the article identifier that sends the visual identification signal; and associating the target object corresponding to the selected identifier ID to the target space position according to the pre-stored correspondence between the identifier ID and the target object.

The item space position correlation unit 802 recognizes the target space position identified by the item identifier that emits the visual recognition signal in the video monitoring screen. Wherein the target spatial location is a spatial location of the item identifier identifying the identified target item.

The article space position association unit 802 obtains, in a video monitoring frame, a target space position identified by the article identifier sending the visual identification signal according to the visual identification signal, specifically, provides the video monitoring frame to a trained image identification model, and identifies the target article identifier sending the visual identification signal in the video monitoring frame; determining the placement position of the marked object associated with the target object identifier according to the position of the target object identifier in the video monitoring picture and the position offset relation between the spatial position of the object identifier and the spatial position of the target object; and taking the placement position as a target space position identified by the object identifier for sending the visual identification signal.

The training process of the image recognition model may refer to the first embodiment, and will not be described herein.

In the apparatus for determining a spatial position of an article according to the second embodiment of the present application, a plurality of article identifiers are included in a video monitoring range, and the article identifier transmitting a visual identification signal is selected by an identifier ID, so that the article identifier having the identifier ID is controlled to transmit the visual identification signal. And a visual identification signal sent by the article identifier can be obtained in the video monitoring picture, and the target space position identified by the article identifier can be determined through the visual identification signal, wherein the target space position is the target space position of the target article identified by the article identifier. The corresponding relation between the identifier ID and the target object is stored in advance, and the information of the target object corresponding to the object identifier with the selected identifier ID is obtained according to the corresponding relation. Thus, the target item is associated to the target spatial location. In the above process, the association relationship between the target space position and the target object enables the identification of the object class stored in each space position to be more accurate through the corresponding relationship between the identifier ID of the object identifier and the object class stored in advance.

Third embodiment

On the basis of the first embodiment, a third embodiment of the present application provides an article non-checkout information prompting method. Fig. 9 is a flowchart of a method for prompting article non-checkout information according to a third embodiment of the present application. The process of determining the target spatial position of the target object in the third embodiment is implemented based on the method provided in the first embodiment, and thus the process of determining the target spatial position of the target object in the third embodiment may be described with reference to the related description of the first embodiment. The article no-checkout information presentation method shown in fig. 9 includes steps S901 to S903.

Step S901, controlling an article identifier having a selected identifier ID within a video monitoring range to transmit a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; and according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position.

The step is used for establishing an association relationship between the target object and the target space position. By the mode, all articles in the video monitoring picture and the space positions corresponding to the articles can be built in a relation one by one; namely, the effect that the stored article can be known from the spatial position of the video monitoring picture can be achieved; for example, from a certain shelf location (target space location), the item stored therein can be known as a commodity of a certain SKU. In this way, in the following step of user behavior tracking, the specific object related to the user behavior can be deduced according to the spatial position related to the user behavior identified in the video monitoring process.

For a detailed description of the process of determining target spatial location information for a target item identified by an item identifier, reference is made herein to the scene embodiment and the first embodiment.

Step S902, if a target user is detected to be located in the video monitoring range in a video monitoring picture, if an intersection point exists between the position of a human body key point of the target user and an article detection frame of the target article and the article detection frame moves from the target space position to an area outside the video monitoring range, adding information of the target article into an article taking list of the target user; and determining the information of the target object according to the target object associated with the target space position.

The method is used for determining whether the user takes the target commodity or not according to the fact that whether the intersection point exists between the user and the target commodity and the moving track of the target commodity is detected in the video monitoring picture.

If an intersection point exists between the position of the key point of the human body of the user and the object detection frame of the target object, the user is judged to be in the process of taking the target object. If the object detection frame of the target object moves from the target space position to an area outside the video detection range, in other words, the moving track of the target object disappears in the video monitoring picture, the user is judged to have taken the target object, and the invoice information of the user is updated.

Step S903, determining whether the target article in the article taking list has checkout record information, if not, determining that the target user has an unsettled risk for the target article, and sending risk prompt information.

The step is used for judging whether the user checks out the target commodity, if not, the checking-out prompt information is sent, and the prompt information can be sent to staff, users and the like. The specific judging process is shown as 203 in fig. 2, and the full-field track of the customer is tracked; (judgment target client) is present in the export/checkout area; comparing the pick (of the target customer) with the checkout bill; and (5) carrying out early warning of the risk of not settling according to the comparison result.

The step of judging whether the target object in the object taking list has checkout record information or not, if not, judging that the target user has non-checkout risk aiming at the target object, and can be realized by the following steps:

Judging whether the objects in the object taking list have checkout record information or not when the target user is in the target area in the video monitoring picture, and if not, determining that the user has non-checkout risk for the objects in the object taking list; the target area is an area where the user performs settlement of the article or an area where the user has completed settlement of the article. The target region may include a first region and a second region, as described below in connection with fig. 6.

tracking the full-field track of the customer in real time (S601), so as to acquire the mobile position information of the target user in the video monitoring picture; and monitoring that the user is in a first area in the video picture, and the moving track of the user meets a certain condition (such as 'appearing at an exit and not passing through a checkout area' in S602 in fig. 6), judging whether the target user takes a bill and a transaction record (S603), and if so, determining that the target user has a complete non-checkout risk for the articles in the article taking bill (S604).

Through the two conditions, whether the target user checks out the target object is judged, and if the target object is not checked out, the risk early warning of the non-checking out is sent. According to the method provided by the embodiment of the application, the corresponding relation between the target space position marked by the object marker and the target object is determined through the visual identification signal in the video monitoring picture, so that the object taken by the user can be determined through the identification of the space position in the video monitoring picture, and the identification accuracy of the target object taken by the user is effectively improved. On the basis, the contact track of the user and the target object and the tracking of the user behavior track in the video monitoring picture are used for monitoring and tracking the user and the target object, so that whether the user finishes checkout on the target object can be accurately determined.

Fourth embodiment

Referring to fig. 10, a logic frame diagram of an article non-checkout information prompting system according to a fourth embodiment of the present application is provided, and the system according to the present embodiment is described in detail below with reference to fig. 10. The embodiments referred to in the following description are intended to illustrate the method principles and not to limit the practical use. Wherein the article non-checkout information prompting system provided by the fourth embodiment corresponds to the scene embodiment and the third embodiment, and therefore, the relevant part can refer to the scene embodiment and the third embodiment. The article non-checkout information presentation system 1000 shown in fig. 10 includes: the article spatial position recognition unit 1001, the article pickup recognition unit 1002, and the article non-checkout risk recognition unit 1003.

The article space position identifying unit 1001 is configured to control an article identifier having a selected identifier ID within a video monitoring range to transmit a visual identifying signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; and according to the pre-stored corresponding relation between the preset identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position.

Wherein the item space position identification unit 1001 corresponds to part 201 in fig. 2. The item space position identification unit 1001 establishes an association relationship between the target item and the target space position. By the mode, all articles in the video monitoring picture and the space positions corresponding to the articles can be built in a relation one by one; namely, the effect that the stored article can be known from the spatial position of the video monitoring picture can be achieved; for example, from a certain shelf location (target space location), the item stored therein can be known as a commodity of a certain SKU. In this way, in the following step of user behavior tracking, the specific object related to the user behavior can be deduced according to the spatial position related to the user behavior identified in the video monitoring process. For example, in a market scene, through a user's operation of taking a target object in a video monitoring screen, detailed information (the core is SKU information, and after SKU information is obtained, the rest of information can be obtained from a database) of the target object can be determined through a target spatial position of the target object in the video monitoring screen.

The article taking and identifying unit 1002 is configured to monitor in a video monitoring screen that a target user is located in the video monitoring range, and if an intersection point exists between a position where a human body key point of the target user is located and an article detection frame of the target article, and the article detection frame moves from the target spatial position to an area outside the video monitoring range, add information of the target article to an article taking list of the target user; and determining the information of the target object according to the target object associated with the target space position.

The article taking and identifying unit 1002 is configured to determine whether the user takes the target article according to whether an intersection exists between the user and the target article and a movement track of the target article detected in the video monitoring screen. The detailed description thereof may refer to the description of fig. 5.

The article non-checkout risk identification unit 1003 is configured to determine whether there is checkout record information on a target article in the article taking list, and if not, determine that the target user has non-checkout risk for the target article, and send risk prompt information.

The article non-checkout risk identification unit 1003 is configured to determine whether the user is checkout a target article, and if not, send a non-checkout prompt, which may be sent to a staff member, the user, or the like. The specific judging process is shown as 203 in fig. 2, and the full-field track of the customer is tracked; (judgment target client) is present in the export/checkout area; comparing the pick (of the target customer) with the checkout bill; and (5) carrying out early warning of the risk of not settling according to the comparison result.

The detailed description thereof may refer to the description of fig. 6.

The article non-checkout risk identification unit 1003 is specifically configured to: tracking the full-field track of the customer in real time (S601), so as to acquire the mobile position information of the target user in the video monitoring picture; and monitoring that the user is in a first area in the video picture, and the moving track of the user meets a certain condition (such as 'appearing at an exit and not passing through a checkout area' in S602 in fig. 6), judging whether the target user takes a bill and a transaction record (S603), and if so, determining that the target user has a complete non-checkout risk for the articles in the article taking bill (S604).

The article non-checkout risk identification unit 1003 is specifically configured to: tracking the full-field track of the customer in real time (S601), so as to acquire the mobile position information of the target user in the video monitoring picture; monitoring that the target user is in a second area in the video monitoring picture (S606), wherein the second area is an area where the target user performs settlement of the articles, such as a settlement area of a mall; when the target user checkout transaction is finished, the system compares the object taking list of the target user with the checkout list (S607), judges whether the corresponding checkout record information exists in the checkout record list for the target object in the object taking list, and determines that the target user has an unclosed risk for the target object if the situation exists that the target object in the object taking list does not exist in the checkout list (S608).

Through the two conditions, whether the target user checks out the target object is judged, and if the target object is not checked out, the risk early warning of the non-checking out is sent. According to the system provided by the embodiment of the application, the corresponding relation between the target space position marked by the object marker and the target object is determined through the visual identification signal in the video monitoring picture, so that the object taken by the user can be determined through the identification of the space position in the video monitoring picture, and the identification accuracy of the target object taken by the user is effectively improved. On the basis, the contact track of the user and the target object and the tracking of the user behavior track in the video monitoring picture are used for monitoring and tracking the user and the target object, so that whether the user finishes checkout on the target object can be accurately determined.

Fifth embodiment

A fifth embodiment of the present application provides an electronic device, and since the electronic device embodiments are substantially similar to the method embodiments, the description thereof is relatively simple, and details of related technical features should be referred to the corresponding descriptions of the method embodiments provided above, and the following descriptions of the electronic device embodiments are merely illustrative. The electronic device embodiment is as follows: fig. 11 is a schematic diagram of an electronic device according to a fifth embodiment of the present application.

As shown in fig. 11, the electronic device provided in this embodiment includes: a processor 1101 and memory 1102, a communication bus 1103 and a communication interface 1104. The processor 1101 is configured to execute the one or more computer instructions to implement the steps of the method embodiments described above. The memory 1102 is used to store one or more computer instructions for data processing. The communication bus 1103 is used for connecting the processor 1101 and the memory 1102 mounted thereon. The communication interface 1104 is used to provide a connection interface for the processor 1101 and the memory 1102.

In the above embodiments, a method for determining a spatial position of an article, an apparatus and an electronic device corresponding to the method are provided, and in addition, a computer readable storage medium for implementing the method for determining a spatial position of an article is provided in embodiments of the present application. The embodiments of the computer readable storage medium provided in the present application are described more simply, and reference should be made to the corresponding descriptions of the above-described method embodiments, the embodiments described below being merely illustrative.

The computer readable storage medium provided in this embodiment stores computer instructions that, when executed by a processor, implement the steps of the method embodiments described above.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

1. Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include non-transitory computer readable media (transmission media), such as modulated data signals and carrier waves.

2. It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. It should be noted that, in the embodiments of the present application, the use of user data may be involved, and in practical applications, user specific personal data may be used in the schemes described herein within the scope allowed by applicable legal regulations in the country where the applicable legal regulations are met (for example, the user explicitly agrees to the user to actually notify the user, etc.). It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

Claims

1. A method of determining the spatial location of an item, comprising:

controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal;

in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal;

and according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position.

2. The method of claim 1, wherein controlling the item identifier within the video surveillance zone having the selected identifier ID to transmit a visual identification signal comprises:

transmitting an opening instruction for opening a visual identification signal to object identifiers in a video monitoring range, wherein the opening instruction comprises a selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction.

3. The method according to claim 1, wherein the obtaining, in the video monitoring screen, the target spatial location identified by the object identifier that transmits the visual identification signal according to the visual identification signal includes:

Providing the video monitoring picture for a trained image recognition model, and recognizing a target object identifier which sends a visual recognition signal in the video monitoring picture;

determining the placement position of the marked object associated with the target object identifier according to the position of the target object identifier in the video monitoring picture and the position offset relation between the spatial position of the object identifier and the spatial position of the target object;

and taking the placement position as a target space position identified by the object identifier for sending the visual identification signal.

4. A method according to claim 3, wherein said identifying the target item identifier in the video surveillance screen that transmitted the visual identification signal comprises:

identifying an item identifier contained in the video monitoring picture;

generating a corresponding partial graph of the object identifier according to each identified object identifier;

in the article identifier partial map, an article identifier that transmits a visual identification signal is identified as the target article identifier.

5. The method of claim 4, wherein the image recognition model comprises a first recognition model, a second recognition model;

The first identification model is used for identifying object identifiers contained in the video monitoring picture;

the second identification model is used for identifying the object identifier sending the visual identification signal in the object identifier partial graph.

6. The method of claim 5, wherein the second recognition model is obtained by training a selected initial visual recognition signal detection model by:

obtaining a data sample set for training the initial visual identification signal detection model, wherein the data samples in the data sample set comprise marked object identifier pictures, and signal transmitting points for transmitting visual identification signals are marked by using a first target frame;

and providing the data sample set for the initial visual recognition signal detection model to train so as to enable the data sample set to meet the training requirement of recognizing the signal emission points in the object identifier local graph, thereby obtaining the second recognition model.

7. The method of claim 6, wherein said providing said data sample set to said initial visual recognition signal detection model for training to meet training requirements for recognition of signal emission points in said partial view of said item identifier, comprises:

Providing the marked article identifier picture for the initial visual identification signal detection model to obtain an identification result output by the initial visual identification signal detection model, wherein the identification result comprises a first prediction frame aiming at the position of a signal emission point of the marked article identifier picture;

obtaining a first loss value according to the difference between the first prediction frame in the identification result and the first target frame in the marked article identifier picture;

adjusting parameters of the initial visual recognition signal detection model according to the first loss value;

repeating the steps until the first loss value reaches the requirement of a preset threshold value, and obtaining a second recognition model after training is completed.

8. The method as recited in claim 6, further comprising: labeling a first target frame on an article identifier picture which transmits a visual identification signal by adopting the following method:

acquiring an article identifier picture for transmitting a visual identification signal;

performing binarization processing on the object identifier picture to obtain a binarized picture;

denoising other information except the signal emission point in the binarized picture to obtain a denoised picture containing the signal emission point;

And adopting a marking frame with a preset shape to frame the signal emission point in the denoising picture to obtain the first target frame.

9. An apparatus for determining the spatial location of an item, comprising: a control unit, an article space position association unit;

the control unit is used for controlling the object identifiers with the selected identifier ID in the video monitoring range to send visual identification signals;

the article space position association unit is used for acquiring a target space position identified by the article identifier for transmitting the visual identification signal according to the visual identification signal in a video monitoring picture; and associating the target object corresponding to the selected identifier ID to the target space position according to the pre-stored correspondence between the identifier ID and the target object.

10. The device according to claim 9, wherein the control unit is specifically configured to send an on command for starting a visual identification signal to an item identifier within a video monitoring range, the on command including a selected identifier ID; and the article identifier with the selected identifier ID sends a preset visual identification signal in a preset time based on the receiving of the starting instruction.

11. A method for prompting a non-checkout of an article, comprising:

controlling an article identifier with a selected identifier ID in the video monitoring range to send a visual identification signal; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position;

if the target user is detected to be located in the video monitoring range in the video monitoring picture, if an intersection point exists between the position of the human body key point of the target user and the object detection frame of the target object and the object detection frame moves from the target space position to an area outside the video monitoring range, adding the information of the target object into an object taking list of the target user; the information of the target object is determined according to the target object associated with the target space position;

judging whether the object in the object taking list has checkout record information or not, if not, judging that the object user has non-checkout risk aiming at the object, and sending risk prompt information.

12. An article outstanding information presentation system, comprising: the device comprises an article space position identification unit, an article taking identification unit and an article non-settlement risk identification unit;

the article space position identification unit is used for controlling the article identifiers with the selected identifier ID in the video monitoring range to send visual identification signals; in the video monitoring picture, according to the visual identification signal, acquiring a target space position marked by the object identifier for sending the visual identification signal; according to the pre-stored correspondence between the identifier ID and the target object, associating the target object corresponding to the selected identifier ID to the target space position;

the article taking and identifying unit is used for monitoring that a target user is located in the video monitoring range in a video monitoring picture, and adding information of the target article into an article taking list of the target user if an intersection point exists between the position of a human body key point of the target user and an article detecting frame of the target article and the article detecting frame moves from the target space position to an area outside the video monitoring range; the information of the target object is determined according to the target object associated with the target space position;

The article non-settlement risk identification unit is used for judging whether settlement record information exists in the target article in the article taking list, if not, judging that the target user has non-settlement risk aiming at the target article, and sending risk prompt information.

13. An electronic device, comprising: a processor and a memory; wherein,

the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of any of claims 1-8, 11.

14. A computer readable storage medium having stored thereon one or more computer instructions executable by a processor to implement the method of any of claims 1-8, 11.