RU2785327C1 - Computer vision system in retail - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the field of computer technology. A computer vision system in retail contains a set of video cameras for capturing images connected by a single network with a server. The block for capturing the stream from cameras is located on the server and is configured to independently read frames from the stream of a set of video cameras using a queuing mechanism that buffers the frames necessary for processing, and the block captures each current frame from the stream, arriving at a frequency pre-set on the camera, and if the difference between the two fixed time characteristics exceeds or equals the threshold value, the frame is decoded and buffered into the queue variable, and if the queue variable is overflowing, the frame is skipped. The detector block is an artificial neural network that receives frames from the queue formed by the camera stream capture block and returns the coordinates of the contours of predetermined objects to be detected. Analyzer block configured to compare the detected objects detected by the detector unit with the reference states stored in the database and classify the types of inconsistencies. The block for sending notifications and visualization is configured to send the results of the identified inconsistencies of the analyzer block to the graphical user interface.

EFFECT: technical result is to improve the accuracy of detection of small and closely spaced objects.

3 cl, 8 dwg

Description

FIELD OF TECHNOLOGY

[001] This technical solution generally relates to the field of computing, and in particular to methods and systems of computer vision in retail. The invention is intended for the intelligent collection and analysis of information from surveillance cameras of retail chains, for example, in the cash register area and/or the product presence area.

BACKGROUND OF THE INVENTION

[002] In the FMCG industry, high turnover is the basis for making good profits. Automation of sales processes is the main indicator of efficiency improvement.

[003] There are various electronic systems for servicing the sale of goods, largely automating the process of selling goods, allowing it to be taken into account, maintain databases, read the graphic markings on the product during the sale, correlate it with the database, determine the product from it and automatically generate a receipt for purchase of goods. At the same time, the insufficient degree of automation of the process of selling weighted goods in supermarkets is due to the inability to automatically identify goods that do not have a special graphic marking (barcode) for automatically generating a receipt for payment.

[004] Also a big problem of modern super- and hypermarkets are the numerous queues at the checkouts during the "peak hours" of the mass call of the buyer.

[005] So known from the prior art is an electronic system for servicing the sale of goods, including at least one video camera connected via a unified network to a server, including a receiving unit connected to each other, incoming information from the video camera, a computing module, a base storage unit of these goods, see utility model patent of the Russian Federation No. 94739, published on May 27, 2010.

[006] This system for servicing the sale of goods is the closest in technical essence and achieved technical result and is selected as a prototype of the present invention.

[007] The disadvantage of this prototype is the insufficient degree of automation of the process of selling goods due to the inability to automatically detect goods that do not have a special graphic marking (barcode or QR code), automatically generate a check for payment and make a calculation, as well as register buyers and their actions for exclusion of dishonest actions on the part of buyers (theft of goods).

ESSENCE OF THE TECHNICAL SOLUTION

[008] The technical challenge or technical problem to be solved in this technical solution is the implementation of a computer vision system in retail.

[009] A useful technical effect in the system is achieved by improving the accuracy of detecting small and nearby objects, for example, goods, as well as the accuracy of determining the occurrence of a queue event, which is confirmed by test measurements in pilot projects conducted with competitors in this field of technical solutions.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0011] In FIG. 1 shows an example of the implementation of the computer vision system in retail.

[0012] In FIG. Figure 2 shows an example of the correct installation of cameras with marked checkout areas.

[0013] In FIG. Figure 3 shows the workflow of the queue module.

[0014] In FIG. 4 shows a variant of the implementation of "boxes" for detecting people's heads.

[0015] In FIG. 5 shows an example of the implementation of notifications about the queue in the messenger.

[0016] In FIG. 6 shows a screenshot from the shelf module camera.

[0017] In FIG. 7 shows the workflow of the shelf module.

[0018] In FIG. Figure 8 shows the visualization of the processes in the pre-processing block from left to right: a frame with corrected distortion, cut and straightened shelves after applying tomography, cut shelves are sent to the detector.

DETAILED DESCRIPTION OF THE INVENTION

[0019] Below will be discussed in detail the terms and their definitions used in the description of the technical solution.

[0020] In this invention, the system refers to a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations (actions, instructions), centralized and distributed databases, smart contracts.

[0021] A command processing device refers to an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs), a smart contract, an Ethereum virtual machine (EVM), or the like. An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. The role of a storage device can be, but not limited to, hard disk drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

[0022] A program is a sequence of instructions intended to be executed by a computer control device or command processing device.

[0023] The invention relates to electronic computing systems for servicing sales in the field of FMCG and can be used in retail chains.

[0024] FMCG - (From the English. Fast Moving Consumer Goods) - the general name for goods of daily consumption by a wide range of buyers, which are relatively cheap and quickly sold;

[0025] PLU-code - (from the English Price-LookUp (PLU)) - a system for accurate identification of goods.

[0026] Dataset - (from the English. Data set) - a set of data.

[0027] POS-terminal (from the English. Point Of Sale - point of sale) - a software and hardware system that is installed at the cashier's workplace.

[0028] DPC - data processing center.

[0029] Server (English server) - an electronic device that performs service functions at the request of the client, providing him with access to certain resources. For the purposes of this description, a server is considered that has a permanent connection to an internetwork that can transmit data to the server from client devices. The server can process this data and send the result of processing back to the client device.

[0030] The internetwork, as well as all connections between all modules and blocks, includes various topologies, configurations and layouts of interconnection components, made with the ability to interconnect corporate, global and local computer networks, and includes, without limitation, traditional wired, wireless, satellite, optical and equivalent network technologies.

[0031] An image or other data input block is a server block that can represent a receiver of incoming signals and a converter for further processing.

[0032] A computing unit is a server unit, which is a microprocessor specially adapted for signal processing.

[0033] A database or database storage unit is a server unit that stores data corresponding to a given data unit, which can be implemented as a hard disk drive, or as flash memory, which refers to semiconductors electrically reprogrammable memory. However, some modules can be combined in separate implementations. For example, different databases can be stored in the same memory module.

[0034] A customer's client device is any form of computing platform connected to a network, such as a internetwork, and capable of interacting with application programs. Typical examples of individual client devices include, but are not limited to, desktop and portable personal computers, "intelligent" cellular phones, (in English - smartphone), portable computers, including tablets, lightweight clients, workstations, "dumb" terminals, connected with the application server, as well as their various layouts and configurations, that is, both physical devices for interacting in communication interaction systems, and virtual devices implemented on programmable computer devices and having a software interface for implementing communication functions.

[0035] Neural network - (artificial neural network) - a mathematical model, as well as its software or hardware implementation, built on the principle of organization and functioning of biological neural networks - networks of nerve cells of a living organism. Neural networks are not programmed, but are trained when the coefficients of connections between neurons are found. In case of successful training, the network will be able to return the correct result based on the data that was missing in the training sample, as well as incomplete and / or “noisy”, partially distorted data. [0036] Near field communication, NFC (“near field communication”, “near contactless communication”) is a short-range wireless high-frequency communication technology that allows the exchange of data between devices located at a distance of about 10 centimeters.

[0037] A QR code (from the English "quick response" means "quick response") or a barcode, a matrix code (two-dimensional barcode) that encodes any information, be it plain text or a link to a website.

[0038] As shown in FIG. 1, the system may include the following implementation examples: queue module, shelf and price tag module.

[0039] It is worth noting that the claimed technical solution can be implemented as, for example, a queue module or as a shelf and price tag module, or include implementations of two or more of these modules at once. This is an application application of the claimed computer vision system.

[0040] Each module performs its tasks, has its own source of information, the results of the two modules can be accumulated in a common graphical user interface (GUI) so that a store employee or an artificial intelligence algorithm can at any time receive visual and information about the general status of the store in the context of events at cash registers and shelves or used separately.

[0041] The modules of queues and shelves are technically organized as docker containers, communication between containers is carried out using POST requests. In a specially designated container for computing on the GPU (graphic processor), neural networks are implemented, there is a container for computing tasks on the CPU (central processing unit) and interaction with the database, and there is also a container that contains the frontend part of the system.

[0042] Consider each module separately.

[0043] The queuing module is designed to count people in the checkout area of the store and send notifications about the queues from the computer to the telegram chat, to the mail of the store director or in the form of a sound signal in the store. There are restrictions on the number of recommended cameras per server, they depend on the characteristics of the video card used. For example, for the GTX 1060 video card, the maximum number of cameras is 30, if there are more cameras, the real-time mode will not be observed during processing, which will lead to the appearance of false positives of the module. There are no restrictions on the choice of camera models. For the correct operation of the queue module, it is necessary to comply with the conditions for mounting the cameras. With correct installation, each camera will have one checkout zone in the field of view, the head of each person in the zone will very rarely intersect with other heads or obstacles, that is, people will not block each other on the video from the camera, and will not hide behind obstacles. On FIG. 2 shows examples of correct installation.

[0044] The checkout area includes two regions that are marked manually by the system administrator in advance (Fig. 3): the cashier's workplace and the area of customers - the area where the threshold number of people is located, at which the store staff will be alerted about the presence of a queue and recommendations for opening cash desks . The threshold number of people can be any, set in the module settings (default - 5). Information from the cameras is captured at a rate of one frame/s in the stream capture block. This frequency is optimal for implementing real-time mode with a large number of cameras in the store (more than 10) and at a single camera processing speed of 30 fps (on NVIDiA 1060, on NVIDIA 1080ti 60 fps), the frequency cannot be changed. The general workflow is shown in Fig. 3.

[0045] The camera stream capture block uses the ffmpeg library to capture the stream. For independent reading of frames from a stream (in order to avoid hanging the stream itself), a queuing mechanism is used, which buffers the frames necessary for processing. The process is split into two child processes:

1) reading frames (writer);

2) frame processing (reader).

[0046] The writer captures every available frame (the current frame from the stream, arriving at the fps rate set on the camera; the camera must be set to more than 1 frame / s), and if the difference between the two fixed timings (for example, unix- times t1 = 1654607278, t2 = 1654607279, difference = 1 s) exceeds or equals the threshold value (in order to fix a certain number of frames per unit of time, in our case - frame per second), the frame is decoded using the ffmpeg library algorithms and buffered into the queue variable, the frame integrity characteristic ("broken" or "not broken" frame, characteristic value 0 or 1, respectively) and the time at which this frame was received are also added there. If the queue variable is full (by default, the queue size is 10 frames) - the frame is skipped, that is, the frame is not added to the queue variable, but is expected when space is freed in it (when the reader reads).

[0047] There is a mechanism for checking the stream for a malfunction: if 30 frames were received in succession that were “unreadable” (broken), the function that reads the stream is restarted.

[0048] The reader is in standby mode. When data appears in the queue, the reader reads the frame and sends it for processing (detector block, etc.). After the frame is read by the reader and sent for processing, the process again goes into standby mode until new data appears.

[0049] The detector block is a Scaled YOLO v4 neural network that returns the coordinates of "boxes" (rectangles whose sides are parallel to the sides of the image - Fig. 4) of detected objects.

[0050] In the queue module, the objects to be detected are human heads. This architecture compares favorably with competitors (SSD, Faster R-CNN, EfficientDet) due to the combination of high processing speed due to the fact that it is a single-stage (one-stage, single-pass) detector, it uses network resolution more efficiently than other single-stage networks. High processing speed is necessary when there are a large number of cameras and real-time mode and accuracy are required. The training of the neural network takes place in strict accordance with the recommendations described by the developers of this network. An important step is the selection of the sizes of anchor boxes ("anchors"), they are selected using the k-means clustering algorithm, in total 9 "anchors" (basic boxes) are selected, which must be distributed over three levels of the network scale pyramid. The "basic option" is three for each scale, ranked so that on the large scale there are small anchor sizes, and on the small scale there are large ones. In the process of learning the first 5 epochs, a decision is made about the success of the distribution of anchors by levels (the criterion is the percentage of changed boxes of the training sample, rewritten boxes, if more than 1% of them - the anchors are unsuccessfully distributed), in case of failure, the distribution of anchors by levels changes.

[0051] In the analytics block, detections related only to the area where buyers are located are filtered (only those boxes of goals whose centers are located in the area are selected), a decision is made about the occurrence of a queue, its duration is determined, and, if necessary, "gluing" two queues in one by the condition of not exceeding the threshold in time between two queues.

[0052] The algorithm for determining the queue in the analytics block works as follows: a sliding window of 20 seconds is used, in which the number of frames with the number of people exceeding a certain threshold set by the user (5 people by default) is counted if the number of such frames exceeds half the window , then we consider that the queue has started, otherwise we consider that the queue has ended. The window moves in increments of 5 seconds, the start time of the queue is considered the moment of the first frame with the number of people exceeding the threshold. Similarly, we determine the end of the queue (the first frame with the number of people less than or equal to the threshold value). The queue is extended if it also exists in the next window (the end of the queue is carried forward). Queues are "glued together" if less than 40 seconds have passed between two queues (the threshold is selected empirically, you can change it in the settings). Sending a notification about the existence of a queue occurs according to two threshold timings of the queue lifetime (5 minutes by default for the first notification, 7 minutes for the second).

[0053] In the GUI (graphical user interface), you can view the notification feed, you can also configure notifications to be sent to Telegram (Fig. 5), by mail, or as a sound signal in the store. Notifications contain different information depending on the evaluation mode of active cash registers. Three modes are possible:

1) "standard": the cash desk is considered active if at least one person is found in the area where the customers of this cash desk are located;

2) "cashier detector": the cash desk is considered active if there is at least one person in the cashier's area during the queue observation time, the cashier detection algorithm is similar to the queue detection algorithm (sliding window logic);

3) "cash server" (in the presence of cash events): determine the state of the cash register by the corresponding signal from the cash server.

[0054] The standard notification contains a frame of the start of the checkout queue at which the alarm was generated, the number of people in the queue of all checkouts at the time the notification was sent, and information depending on the presets.

[0055] The shelf module is designed to analyze the location of goods and price tags on the shelves of the sales floor racks, correlating the actual location (hereinafter referred to as the realogram) with the planogram. Planogram - the planned arrangement (layout) of goods on store shelves. The planogram is usually updated every month at the store; it can be presented in various retail chains in various forms: xls-tables, pdf-files, etc. Examples of planograms are shown in Fig. 6. Realogram - the actual location of goods on the shelves. For correct operation of the shelf module, cameras with a resolution of at least 8 megapixels are required, as well as compliance with the camera installation conditions. With correct installation, in the field of view of each camera there will be no more than two racks with goods on the shelves in full "length" (Fig. 6).

[0056] Before launching the shelf module into operation, a preliminary configuration of each camera is carried out. Preset Algorithm:

1) the camera settings automatically set the resolution to 3840×2160, the maximum bitrate;

2) the algorithm for correcting radial distortion from the opencv library is launched to remove the "fisheye"; the required coefficients were obtained as a result of calibrating an arbitrary 8 megapixel camera of a specific model with a resolution of 3840 × 2160 set on it in accordance with the recommendations in the opencv documentation for calibrating the camera using a checkerboard (if the camera model changes, a new calibration is required);

3) the shelves are marked with a polygon of four points, which are placed strictly in the corners of the "physical" shelves of the rack (the rack can be long and consist of several standard shelves, usually 125 cm long); shelf layout allows us to align the perspective of the shelf, as well as control the width of the shelf that is then fed to the product detector, so we provide more stable detection and subsequent recognition of products;

4) tomography matrices are formed for each chamber shelf at four marked points, the matrices are written to the database for further use during module operation.

[0057] The operation of the shelf module is shown in FIG. 7.

[0058] The frame capture block uses the ffmpeg library to capture a frame, frames are captured every two minutes and written to the server disk, subsequent processing blocks in accordance with the diagram in FIG. 7 run at the frequency set in the settings (once per hour by default).

[0059] The frame pre-processing block works as follows, sequentially for the frame of each camera:

1) distortion is corrected as described in the presetting algorithm;

2) shelves are cut and an appropriate tomography matrix is applied to each shelf;

3) scaling the image of each shelf to the size of the detector network.

[0060] The detector block is a Scaled YOLO v4 neural network that returns the coordinates of product "boxes", price tags, and the product price area written on the price tag. The advantage of using this architecture is in speed and accuracy, as well as in the efficiency of detecting small, closely spaced objects (this is important when detecting curds, jars of baby food, and the price area on price tags).

[0061] The classifier block receives cut out areas of the product image as input in accordance with the coordinates received from the detector block. The purpose of the classifier block is to recognize the product in the cut out area. The classifier block contains a neural network of the transformer family - swin. The neural network is trained as a classifier (a cross-entropy loss function with softmax is used, optimization by the Adam algorithm with standard settings, batch size 16, augmentations: contrast change, smoothing, optical distortion, perspective change, cutout, normalization with imgenet dataset parameters, pretrained weight on imagnet), then the last classification layer is removed, and then embeddings are used to assess the similarity of products in the images. Thus, the system will be able to recognize a new product only when a template for this product is added to the database via the GUI and it will not be required to retrain the network.

[0062] Product templates are created when the user adds them to the GUI, this happens in several stages:

1) loading an image from the selected camera with graphically selected "boxes" of goods (successive application of the frame preprocessing and detector blocks to the frame);

2) hovering the mouse cursor over the product of interest, selecting the plu number for it (product code or class) from the list provided (it is possible to search the list by name/code);

3) pressing the button to save the template to the database;

4) after adding the template of each product code, the clustering procedure is started, where all templates of this code are clustered into several groups by cosine distance, the averaged template for each group is written to the database and then used when the classifier block is working.

[0063] When the classifier block is launched, product images (cut out "boxes" of goods from the shelf image) are processed by the neural network described above, processing occurs in batches (batch), the product boxes from all shelves of all cameras are combined into batches, the batch size is set depending on computing power of the server on which the calculation takes place. By default, the batch size is 64, which means that in one forward pass, the neural network will process 64 images. The resulting product vectors are then compared by cosine distance with the averaged cluster vectors of each product in the database, and plu code labels are returned to the target product by the averaged vector closest to the target vector. Target product - the product whose plu code we want to know. The plu code is a numerical identifier of the product on the store, for example, 56789123. The larger the cosine distance (closer to 1), the closer the vectors, that is, the "similar" images, so after calculating the distances to the database vectors, a vector is selected that gives the closest possible to 1 distance value, its plu code is assigned to the compared vector (target product vector).

[0064] In addition, the classifier block contains a bidirectional LSTM price digit recognition neural network trained with etc (Connectionist Temporal Classifications) error function on own data. This architecture has clear advantages in accuracy over the well-known FineReader and Tesseract packages. To recognize numbers, only those price boxes that are inside the price tag boxes are used (a preventive measure to exclude false positive price detections in arbitrary places on the shelves).

[0065] The analyzer (analytics) block compares the actual display of goods with the reference one, and also calculates various statistics. The analyzer, using the coordinates of the recognized products, builds them into a sequence of products on the shelf, grouped by individual product code (PLU). According to a pre-agreed protocol, we receive a planogram from the client (that is, we have a reference sequence of goods on the shelves, taking into account their number in width and height). Also implemented is the ability to receive from the client with a certain frequency of information about the price of the goods and its availability in the warehouse of the store. The analyzer compares the actual sequence of goods with the reference one, and divides them into the following categories:

1. According to the planogram - the goods are in "their place".

2. Face error - the product is in its place, but the quantity of the product in width / height does not match the planogram.

3. Shelf error - the product is not on its shelf.

4. Number error - the product is on its shelf, but the serial number of its placement within the shelf is not observed.

5. Not displayed - the product is not on the shelf.

6. No leftovers - the product is not on the shelf, and also not in the store's warehouse.

7. Equipment error - the number of shelves according to the planogram and on the actual shelf in the store is different.

[0066] The analyzer calculates the percentage of compliance with the planogram, which is defined as the ratio of the number of positions "according to the planogram" to the total number of "not set" and the number of "shelf errors". The analyzer compares the price on the price tags found under the product items with the reference price received from the client and displays the relevant information in the application interface.

[0067] The results are visualized in a GUI (graphical user interface), providing an overall picture of the state of affairs in the store regarding key areas that affect customer experience and attractiveness of the store (as a consequence of its business efficiency). Based on the data received: from analyzers (queues, control of planograms, prices on price tags) and from accounting stores (number of employees per shift, number of working cash desks at the moment, availability of specific goods on the balances in the store, current cost of goods at the moment), in The GUI generates visual reports with action recommendations (you need to open an additional cash register or change the price tags in the dairy department) in detail. These recommendations affect operational management and have an immediate effect, but in addition to operational recommendations, reports for the period are generated in the GUI, which provide an opportunity to look at the retrospective and make strategic business decisions. An integral part of recommendations and actions is sending notifications with a certain type of tasks to the user, and since. There are many tasks and for different business processes in the system there is a module for prioritizing tasks (criticality), based on the assigned level of criticality, priority is set for the execution of tasks by store employees. Basically, the system has three levels of criticality according to which tasks are distributed: Normal, Important, Critical. The distribution of tasks by these levels and queuing in the first place (for the first time on a new shift) is done based on the initial data received at the start and the constants set in the system: time, type of task (basically putting the goods on the shelf is more important than correcting the price tag, but there are nuances), the number of performers at the start and for each task, based on its type and level of criticality, time is allocated for execution. Employees do not see all the tasks, they see only the most critical ones at the moment and receive new ones as they are completed. Further, all tasks (both newly created and previously created) exist in a single linked coordinate system, and tasks of one module (for example, Queues) at the moment can become the highest priority over all others, and vice versa, there can be tasks that are created in the morning and will be closed at the end day (end of the shift) automatically by the system with the status "not completed", because the number of priority tasks and the number of resources (employees) for their implementation did not allow performing tasks that would bring less return for the business in the time period of the shift. All tasks have the concept of a life cycle, and within its life cycle, a task can change the level of criticality either to itself (for example, a large queue arose and lasted long enough for the task to become critical, but most people came for short purchases, i.e. with a small number of goods in the check, and the queue quickly reached the standard), or under the influence of other modules (for example, there were voids on the shelves that reached the “Critical” level, but at the moment the situation on the cash line reached a level when the task of opening a new cash desk becomes lowering the level of other tasks as the highest priority, again the system takes into account: the time of day, the number of buyers on the trading floor and at the checkout at the moment, the number of performers at the moment to complete tasks) and most importantly, that tasks can be closed by users in the interface (Mobile application ), but its life cycle will be closed by the system only after checking and confirming by the system itself that the executor effectively completed the task (displayed goods, opened a cash register, etc.).

[0068] As we have already described above, the queue control module and the planogram control module generate signals (tasks) to correct current inaccuracies. Passing through the prioritization module, signals (tasks) are queued for execution. For example, in queues, the level of criticality depends on the number of people standing in line at the same time and the time they stand near the cash register, the more people and the longer they stand, it affects the criticality. In planogram control (layout control), criticality is determined by the amount of goods that are not on the shelf at the same time (while it is in stock) and the size of the voids on the shelves. There is also a common parameter that combines the level of criticality of both modules - this is the time of day, during peak hours it is important that the goods are on the shelf and the “traffic jams” at the exit from the store do not reach a critical nature. Thus, the task prioritization module is a point of control over the priorities and urgency of performing a particular task in the first place. Due to the fact that each store operates in a highly resource-limited mode (lack of staff), it is very important to competently manage this resource effectively, only in this case the store can get the maximum possible result for the operating day. Another detail is also important - the system as a whole allows you to remove the burden of “decision making” from the line staff (where to go first? What task to take now?) and, as a result, reduce stress by directing employees to perform specific tasks.

[0069] This technical solution can be implemented as a computer vision computer system in retail, which contains one or more of the following components:

• a processing component containing at least one processor,

• memory,

• multimedia component,

• audio component,

• input / output interface (I / O),

• sensory component,

• data transfer component.

[0070] The processing component mainly manages all system operations, such as processing data about a user or a request for computer vision, as well as managing display, phone call, data transmission, camera operation, and recording operation of a mobile communication device. The processing component may include one or more processors that implement instructions for completing all or part of the steps from the above methods. In addition, the processing component may include one or more modules for convenient interaction between other processing modules and other modules. For example, the processing component may include a media module for convenient, lightweight interaction between the media component and the processing component.

[0071] The memory is configured to store various types of data to support the operation of the system, such as a database of user profiles. Examples of such data include instructions from any application or method, contact data, address book data, messages, images, videos, etc., all of which run on the system. The memory may be implemented as any type of volatile memory, non-volatile memory, or a combination of both, such as static random access memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, and others, without being limited.

[0072] The media component includes a screen providing an output interface between a system that may be installed on a user's mobile communication device and the user. In some implementations, the screen may be a liquid crystal display (LCD) or a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input from a user. The touchpad includes one or more touch sensors in terms of gestures, touching and sliding on the touchpad. The touch sensor can not only sense the subject's touch boundary or swipe gesture, but also determine the length of time and pressure associated with the touch and slide operation mode. In some embodiments, the media component includes one front camera and/or one rear camera. When the system is in an operating mode, such as shooting mode or video mode, the front camera and/or rear camera can receive media data from outside. Each front camera and rear camera can be one fixed lens optics system or can have focal length or optical zoom.

[0073] The audio component is configured to output and/or input an audio signal. For example, the audio component includes one microphone (MIC) that is configured to receive an external audio signal when the system is in an operating mode such as call mode, recording mode, and speech recognition mode. The received audio signal can be further stored in memory or routed through the data component. In some embodiments, the audio component also includes a single speaker capable of outputting an audio signal.

[0074] An input/output (I/O) interface provides an interface between a processing component and any peripheral interface module. The above peripheral interface module may be a keyboard, steering wheel, button, etc. These buttons may include, but are not limited to, a start button, a volume button, a home button, and a lock button.

[0075] The sensor component includes one or more sensors and is configured to provide various aspects of assessing the state of the system. For example, the sensor component can detect system on/off states, the relative position of components, such as a display and a keypad, of a single system component, the presence or absence of contact between the subject and the system, and the orientation or acceleration/deceleration and temperature changes of the system. The sensor component includes a proximity sensor configured to detect the presence of a nearby object when there is no physical contact. The sensor component includes an optical sensor (eg, CMOS or CCD image sensor) configured to be used in application rendering. In some embodiments, the sensor component includes an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0076] The communication component is configured to facilitate wired or wireless communication between the system and other devices. The system can access a wireless network based on a communication standard such as WiFi, 2G, 3G, 5G, or combinations thereof. In one exemplary embodiment, the data communication component receives a broadcast signal or broadcast related information from an external broadcast control system via a broadcast channel. In one embodiment, the data communication component includes a near field communication (NFC) module to facilitate near field communications. For example, the NFC module may be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0077] In an exemplary embodiment, the system may be implemented by one or more Application-Specific Integrated Circuits (ASICs), a Digital Signal Processor (DSP), a Digital Signal Processor (DSP), a Programmable Logic Unit (PLU), a field programmable logic chip operation (FPGA), controller, microcontroller, microprocessor or other electronic components, and can be configured to implement the method.

[0078] In an exemplary embodiment, the non-volatile computer-readable medium includes a memory that includes instructions, where the instructions are executed by a system processor to implement the computer vision system implementation methods described above. For example, a non-volatile computer-readable medium can be ROM, random access memory (RAM), compact disk, magnetic tape, floppy disks, optical storage devices, and the like.

[0079] The computing system may include a display interface that transmits graphics, text, and other data from the communications infrastructure (or from a framebuffer, not shown) for display on a media component. The computing system further includes input devices or peripherals. Peripheral devices may include one or more devices for interacting with a user's mobile communications device, such as a keyboard, microphone, wearable device, camera, one or more audio speakers, and other sensors. Peripherals may be external or internal to the user's mobile communication device. The touch screen can display typically graphics and text, and also provides a user interface (such as, but not limited to, a graphical user interface (GUI)) through which a subject can interact with the user's mobile communication device, such as accessing and interacting with with applications running on the device.

[0080] The elements of the proposed technical solution are in a functional relationship, and their joint use leads to the creation of a new and unique technical solution. Thus, all blocks are functionally connected.

[0081] All blocks used in the system can be implemented using electronic components used to create digital integrated circuits, which is obvious to a person skilled in the art. Not limited to, microcircuits, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGA), the logic of which is set by programming, can be used. Programmers and debugging environments are used for programming, allowing you to set the desired structure of a digital device in the form of a circuit diagram or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs can be programmable logic controllers (PLCs), basic matrix crystals ( BMK), requiring a factory production process for programming; ASIC - specialized custom-made large integrated circuits (LSI), which are significantly more expensive for small-scale and single-piece production.

[0082] Typically, the FPGA chip itself consists of the following components:

• configurable logical blocks that implement the required logical function;

• programmable electronic links between configurable logic blocks;

• programmable input/output blocks that provide communication between the external output of the microcircuit and the internal logic.

[0083] Blocks can also be implemented using read-only memories.

[0084] Thus, the implementation of all used blocks is achieved by standard means based on the classical principles of implementing the fundamentals of computer technology.

[0085] As will be appreciated by one of skill in the art, aspects of the present technical solution may be implemented as a system, method, or computer program product. Accordingly, various aspects of the present technical solution may be implemented solely as hardware, as software (including application software, etc.), or as an embodiment combining software and hardware aspects, which may be generally referred to as a "module" , "system" or "architecture". In addition, aspects of the present technical solution may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code embodied thereon.

[0086] Any combination of one or more computer-readable media may also be used. The computer-readable storage medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specifically, examples (non-exhaustive list) of a computer-readable storage medium include: an electrical connection using one or more wires, a portable computer diskette; hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or Flash memory), fiber optic connection, compact disc read only memory (CD-ROM), optical storage device, magnetic storage device or any combination of the above. As used herein, a computer-readable storage medium can be any flexible storage medium that can contain or store a program for use by or in connection with a system, device, apparatus.

[0087] The program code embedded in a computer-readable medium may be transmitted using any medium, including, without limitation, wireless, wired, fiber optic, infrared, and any other suitable network or combination of the foregoing.

[0088] The computer program code for performing the operations for the steps of the present technical solution may be written in any programming language or combinations of programming languages, including an object-oriented programming language such as Python, R, Java, Smalltalk, C++, and so on, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may be executed in whole, in part, on the user's computer, or as a separate software package, in part on the user's computer and in part on a remote computer, or entirely on a remote computer. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN), a wide area network (WAN), or a connection to an external computer (eg, via the Internet via ISPs).

[0089] Aspects of the present technical solution have been described in detail with reference to block diagrams, circuit diagrams and/or diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the present technical solution. It should be appreciated that each block from the block diagram and/or diagrams, as well as combinations of blocks from the block diagram and/or diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, a special purpose computer, or other data processing device to create a procedure, such that the instructions executed by the computer processor or other programmable data processing device create the means to implement the functions/actions specified in block or blocks of a flowchart and/or diagram.

[0090] These computer program instructions may also be stored on a computer-readable medium that can control a computer other than a programmable data processing device or other than devices that operate in a particular manner such that the instructions stored on the computer-readable medium create a device including instructions that perform the functions/actions indicated in the block diagram and/or diagram.

Claims (17)

1. Computer vision system in retail, containing: • a set of video cameras for capturing images in the shelf area and in the checkout area, united by one network with the server on which are located: • a block for capturing the stream from video cameras, configured to independently read frames from the frame stream of a set of video cameras, by means of a queuing mechanism that buffers the frames necessary for processing, moreover the block for capturing a stream from video cameras captures each current frame from a stream arriving at a frequency set in advance on video cameras from a set of video cameras, captures time characteristics and, if the difference between two recorded time characteristics exceeds or equals a threshold value, decodes the frame and buffers it into a queue variable , wherein: – if the data appeared in the queue, the frame is sent for processing; – if the queue variable is full, the frame is skipped; – if several bad frames are received in succession, the capture function is restarted; • frame preprocessing block, sequentially for each frame performing: - distortion correction; – application of the homography matrix; – scaling the image to the size of the detector neural network; • a detector block, which is an artificial neural network that receives as input processed frames from the queue formed by the camera stream capture block and returns the contour coordinates of predetermined objects to be detected with their assignment to the corresponding zone; • classifier block, which is an artificial neural network that receives cut out areas of the image of objects as input in accordance with the coordinates received from the detector block, and recognizes objects in the cut out area; • an analyzer unit configured to compare the objects recognized by the detector unit with the reference states stored in the database, and to classify the results of the comparison into categories with the identification of inconsistencies; • block for sending notifications and visualization, configured to send the results of inconsistencies identified by the analyzer block to the graphical user interface. 2. The system according to claim 1, characterized in that the mechanism for checking the stream for a malfunction is implemented: if n frames were received in succession, which are broken, the function that reads the stream is restarted. 3. The system according to claim 1, characterized in that the architecture of the artificial neural network Scaled YOLO v4 is used in the detector unit.

Images