CN115457458A - Non-contact intelligent checking system and method - Google Patents
Non-contact intelligent checking system and method Download PDFInfo
- Publication number
- CN115457458A CN115457458A CN202211023399.6A CN202211023399A CN115457458A CN 115457458 A CN115457458 A CN 115457458A CN 202211023399 A CN202211023399 A CN 202211023399A CN 115457458 A CN115457458 A CN 115457458A
- Authority
- CN
- China
- Prior art keywords
- image
- checking
- goods
- shelf
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q10/0875—Itemisation or classification of parts, supplies or services, e.g. bill of materials
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
- G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Entrepreneurship & Innovation (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a non-contact intelligent inventory system, which combines the service processing flow of remote optical image transmission and a deep learning visual model, traverses the quantity, the type and the coordinate distribution condition of main objects in a detection picture through a target detection model SSDMobilene, segments a goods shelf label image according to the inference result of the detection model, and realizes the quantity statistics and the target comparison retrieval of inventory goods according to the character reading result of an image text recognition network ICRNN. The manual operation in the checking work can be avoided, additional electronic tags or non-contact tags do not need to be added to the commodities, and the remote checking and positioning query of the number of the commodities are realized by arranging the camera.
Description
Technical Field
The invention relates to the field of intelligent inventory management, in particular to a non-contact intelligent checking system and a non-contact intelligent checking method.
Background
The traditional inventory checking work needs to judge the morphological distribution rule of the displayed commodities on the edge, texture and shape according to the visual function of a professional, distinguish the target types, count the quantity and position of the commodities and then accurately record the commodities so as to ensure the accurate control of the information of the inventory commodities. With the continuous expansion of the variety, quantity and scale of inventory goods, when enterprises or merchants need to collect inventory information more quickly and frequently to adjust business strategies, the inventory work in units of individuals is difficult to be performed due to the common influence of factors such as the operating proficiency of workers, the information feedback efficiency and the environment. More and more automated inventory count systems are being developed.
Inventory checking systems, which are common in the prior art, include the following:
firstly, a non-automatic checking system based on a handheld checking machine needs a manual handheld machine to check commodities and is matched with a management system to realize daily management; the checking application checking system based on the checking machine needs to add a bar code or an electronic tag corresponding to the identification identity to each commodity, a large amount of hardware and maintenance cost need to be paid in the management process, and the radio frequency signal transmission process of the checking machine is easily interfered by metal on a storage environment, a shelf or materials.
Second, unmanned-vehicle-driven automatic inventory systems, often combined with inventory machines or rfid, template matching technologies. In the checking system based on image transmission processing, the non-artificial intelligent visual algorithm generally searches for the commodity in a template matching mode, and the recognition accuracy and the generalization of the pictures in different backgrounds are poor.
Thirdly, the inventory checking system based on the artificial intelligence method can distinguish the types and the quantity of the stored goods basically through image transmission and a visual algorithm, but the functions are incomplete, the affiliation relationship between the goods and the information of the goods shelf to which the goods belong cannot be automatically identified, and a method for distinguishing different goods shelves is not involved. Although the artificial intelligence image processing method can realize the classified statistics of goods on shelves, the number of shelves and the tag information cannot be automatically analyzed according to the information in the picture, and the attribution processing between the goods and the goods on different shelves cannot be realized.
Disclosure of Invention
Aiming at the existing problems, the invention provides a non-contact intelligent checking system by combining remote optical image transmission and visual processing algorithm, aiming at the problems of fast counting, searching and positioning management of a plurality of commodities in a large quantity by checking work, the invention can automatically identify the number of the commodities in a multi-storeroom storage field, and simultaneously, automatically constructs a management system matched with the characters of the label plate of the goods shelf, thereby constructing a high-efficiency and accurate multifunctional checking system.
The core thought of the invention is as follows: the commodity category and the label plate character information in the picture are comprehensively read by combining two artificial intelligent visual models, the commodities and the number of shelves in the field picture are automatically identified through a visual algorithm, and the positioning, attribution and query functions of the shelves and the commodities are realized according to the identification number and the coordinate position of the label plate of the shelf.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
a non-contact intelligent checking system is characterized by comprising a basic information module, an image acquisition module, a checking service module and a visual management module;
the basic information module is used for pre-configuring basic information required by system operation;
the image acquisition module is used for acquiring field pictures stored in the storage shelf and preprocessing the acquired images;
the checking service module is used for checking the quantity of the goods on the storage shelf or retrieving the goods according to the retrieval words and storing the obtained checking result or retrieval result into the database;
and the visual management module is used for visually displaying the monitoring pictures and the checking results of the storage goods shelves.
Furthermore, the image acquisition module comprises a camera, an image transmission unit and an image preprocessing unit;
the camera and the remote image transmission unit are used for shooting the on-site goods shelf through the monitoring camera on the storage goods shelf and remotely transmitting the shot images to the inventory business module;
and the image preprocessing unit is used for carrying out preprocessing operation on the shot image.
Furthermore, the checking service module comprises a goods traversing checking and searching unit and an automatic processing and storing unit of the result;
the goods traversing checking and searching unit is used for checking the number of the goods and the number information of the goods shelf where the goods are located based on the target detection model and the image text recognition network, or retrieving related goods information according to the retrieval key words;
and the result automatic processing and storing unit is used for automatically outputting the checking or searching result to the visual management module and storing the result into the database.
Further, the target detection model adopts an SSDMobilene deep learning model and is used for detecting the quantity, the type and the coordinate distribution information of the objects in the monitoring picture.
Further, an image text recognition network adopts an ICRNN model, the ICRNN model is improved based on a CRNN model, a channel attention mechanism is introduced into the CRNN model, and a residual error network module is used for replacing a traditional CNN layer and recognizing characters on a label plate of a shelf to obtain the shelf number.
Further, the basic information module comprises a model loading and parameter defining unit and a database configuration and field management unit;
the database configuration and field management unit is used for defining and inputting the fields of the data forms in the database;
and the model loading and parameter defining unit is used for configuring the target detection model and the hyper-parameters of the image text recognition network.
Further, the visual management module comprises a monitoring image preview unit and an AI analysis image display unit;
the monitoring image previewing unit is used for previewing the monitoring image shot by the camera;
and the AI analysis image display unit is used for displaying the AI analysis picture into a specified function area through a plurality of function windows, and executing various operations through the operation buttons in the function area.
A non-contact intelligent checking method is characterized by comprising the following steps:
step 1: clicking a goods checking button or a goods searching button in the functional window, if the goods checking is selected, turning to the step 2, and if the goods searching is selected, turning to the step 3;
step 2: inventory statistics of goods
Step 201: selecting a camera in a storage site, and selecting a shelf image in a picture shot by the camera;
step 202: setting a coordinate interval of an inventory range in the obtained shelf image;
step 203: carrying out target detection in the range of the coordinate interval by using a target detection model, outputting a detection result, and segmenting the detection result to obtain a shelf label image;
step 204: inputting the obtained shelf label plate image into an image text recognition network to obtain a character text on the label plate, and obtaining a shelf number according to the character text on the label plate;
step 205: obtaining the shelf number of the commodity shot on the shelf according to the shelf number, repeating the steps 201-204, and finally counting to obtain an inventory result, wherein the inventory result comprises the number and the type of each commodity and a shelf number list;
and 3, step 3: cargo location retrieval
Step 301: searching in the checking result according to the input retrieval key word;
step 302: and outputting the quantity of the commodities and the shelf numbers corresponding to the search keywords.
The beneficial effects of the invention are:
compared with an inventory machine, the remote inventory machine has the advantages that non-contact remote inventory operation is basically realized in space, a large number of workers do not need to go to the site to perform work such as checking, counting and the like, and the number of the workers can be greatly reduced. In the aspects of data statistics, input and maintenance, the system realizes data connection through python and MySQL data and a network transmission tcp protocol, realizes data interaction with a GUI system, obtains the checking result through reasoning and calculation of a visual model, does not need manual input, and avoids misjudgment risks of manual operation. On the aspect of operation performance, the system combines a mature target detection model and a reasonably improved image text recognition model to carry out work, and has obvious advantages on processing efficiency. In addition, the invention combines a working model of target detection and image text recognition, can automatically distinguish, classify and count warehouses with different commodities and goods shelves on site (distinguish and recognize the goods shelves at the same time), and improves the deployment flexibility in actual inventory checking application.
In conclusion, the system can get rid of manual operation in the checking work, the commodities are counted and positioned through the AI image technology, additional electronic tags or non-contact tags do not need to be added to the commodities, and the checking and positioning query of the number of the commodities can be remotely carried out by operating on the operating system after the camera is deployed.
Drawings
FIG. 1 is an architecture diagram of the system of the present invention;
fig. 2 is a flowchart illustrating an inventory statistics of an inventory management window of the system according to the present invention;
FIG. 3 is a flowchart of the system for retrieving merchandise in the retrieval window;
FIG. 4 is a diagram of a basic information preview window of the system of the present invention;
FIG. 5 is a schematic diagram of an inventory function area of the system of the present invention;
FIG. 6 is a schematic diagram of an inventory analysis of the system of the present invention;
FIG. 7 is a diagram illustrating the operation result of the checking program of the system according to the present invention;
FIG. 8 is a schematic diagram of a search function area of the system of the present invention;
FIG. 9 is a schematic diagram of a search analysis of the system of the present invention;
FIG. 10 is a diagram illustrating a visual text search result of the system of the present invention;
FIG. 11 is a schematic diagram of the performance test results of the invention after business inference is performed on samples collected randomly in batches in a development model;
FIG. 12 is a schematic view of the attention distribution of a feature channel;
FIGS. 13a-13b are respective normal and residual network block structures;
FIGS. 14a-14b are graphs showing the variation of the recognition accuracy of ICRNN and CRNN, respectively;
FIGS. 15a-15b are loss curves for ICRNN and CRNN, respectively.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the following further describes the technical solution of the present invention with reference to the drawings and the embodiments.
1. Non-contact intelligent checking system composition
The system comprises a DS-IPC-B12HV2-IA remote monitoring camera of Haikangweiyin (HIKVISION), a computer server, an loving (AIDU) JX2401 display, a Graphic Processing Unit (GPU) and a shelf with a label. The system realizes data transmission tcp protocol through the network through the python and MySQL databases, and communicates with the graphic operation management system and performs data interaction.
The software structure of the system is shown in fig. 1, and comprises basic information, image acquisition, inventory service and a visual management module;
the system comprises a basic information module, a database configuration and field management unit, a data processing module and a data processing module, wherein the basic information module is used for carrying out advanced configuration on basic information required by software operation and comprises a model loading and parameter definition unit and a database configuration and field management unit, and the database configuration and field management unit is used for field definition and data entry of a data form in MySql; the model loading and parameter defining unit is used for configuring network models and hyper-parameters related to the deep learning network on which the system function is realized;
the image acquisition module is used for shooting the on-site pictures of the storage (goods shelf) and receiving, storing and preprocessing the images of the remote monitoring camera; the remote monitoring camera is used for shooting the pictures of a storage (goods shelf) site; the remote image transmission unit is used for remotely transmitting the shot image to the system; the image preprocessing unit is used for preprocessing the received images, and mainly performs normalization processing on the shot warehousing site images on three R, G and B color channels uniformly, wherein the normalized applicable mean values are respectively [0.4914,0.4822 and 0.4465], and the applicable variances are respectively [0.2023,0.1994 and 0.20103];
the inventory service module is used for pushing and synchronously storing an obtained result to a database after the functional service is realized based on inventory or retrieval result information obtained by a heavy-load deep learning model and relevant service logic; the system comprises a goods traversal checking and searching unit and an automatic processing and storing unit of results, wherein the goods traversal checking and searching unit is used for checking goods or retrieving related goods quantity and shelf number information according to retrieval keywords; the result automatic processing and storing unit is used for automatically processing (outputting statistical text notification) the checking or searching result and storing the result in a database;
the visual management module is used for displaying the monitoring picture and the AI analysis picture on site in a visual mode and displaying the monitoring picture and the AI analysis picture in a function area appointed by the system; the device comprises a monitoring image previewing unit and an AI analysis image display unit, wherein the monitoring image previewing unit is used for previewing a monitoring image shot by a monitor; the AI analysis image display unit is used for displaying an AI analysis picture in a designated functional area;
the human-computer interaction interface of the checking system is divided into functional windows such as site preview, checking management, commodity retrieval and the like according to the main body function, and a user can select to perform checking or retrieval functions on shelf pictures in a certain or all camera picture areas on the site according to requirements. Through the operation buttons of the function window, on-site monitoring pictures, commodity catalog parameter tables, shelf catalog parameter tables, inventory AI checking pictures, inventory checking result tables, inventory AI retrieval pictures, target retrieval result tables and corresponding text results can be obtained in the GUI system.
For data information obtained in the running process of the checking system, the text is managed by uniformly using the MySql database, the management system using the database has the characteristics of convenience in operation, small redundancy, high independence and clear structure, and results obtained by the operation of the functional window can be updated to the MySql at any time.
2. Technical principle of the system
The system is mainly designed by combining the service processing flow of remote optical image transmission (video stream reading of a monitoring camera and OpenCv) and a deep learning visual model (SSDMobilene and ICRNN).
The ICRNN is an image text recognition model improved on the basis of a traditional CRNN model, and is responsible for a second-stage task of an OCR model and used for reading character text information containing character images. In the design of the convolution backbone network, the early CRNN uses the VGG-16 model with batch normalization, but the VGG network has a simple structure and a large parameter amount, so that the invention designs the ICRNN model based on the residual error neural network and the attention mechanism improvement. The CRNN network needs to extract and combine and transfer the spatial distribution features and the time series features of the input image step by step, so that the network includes two modules, namely a convolutional neural network and a cyclic neural network. The ICRNN and the CRNN models constructed by the method have the same cycle network stage structure, and both use the serially connected bidirectional LSTM layers as a time sequence characteristic extraction network. When designing a convolution network, the ICRNN replaces the traditional CNN layer in the VGG with a residual network block, and uses a channel attention module without changing the output shape to give weight to the characteristic channel, thereby improving the convolution network structure of the CRNN. The convolution extraction layer based on the residual module and the attention mechanism can more comprehensively characterize the image and correct the bottom layer error in the transmission process. In a real storage site where external environment changes or the label character format is irregular, the ICRNN combined with the residual mapping network and the attention mechanism can be used for better reducing recognition errors and obtaining better generalization performance. Table 1 is a description of the structural parameters of ICRNN.
TABLE 1
The invention considers the complex background of the storage environment, the challenging performance of factors such as the change of illumination conditions, different monitoring angles of a camera and the like on the identification task, improves the image feature extraction network of the CRNN and designs a new deep convolution layer. Network models of various structures have been introduced in the research process of CNNs to enhance recognition accuracy in different scenarios, but the processing of the same batch of features by the conventional network layer is equivalent regardless of increasing network depth and complexity. The design of the attention mechanism can simulate the distinguishing reaction mechanism of the human visual system to different significant pixel regions, and weaken the influence of irrelevant pixel regions such as the background on the recognition result in the feature extraction process, as shown in fig. 12.
For an input feature diagram, two 1x1 convolutional networks are used in a channel attention mechanism to distribute weight factors representing attention coefficients to the dimension of an output channel of the feature diagram, so that a model can feed back according to the influence on loss values on different channels and readjust the attention weights distributed on different channels, the influence of noise and non-key information on a result is effectively inhibited by the model in a training process, the convergence efficiency of the model is improved, and the calculation process is as follows:
A c (IF)=σ(Knernel(AvgPool(IF)+
Knernel(MaxPool(IF)))
wherein: IF is the incoming feature map, avgPool and MaxPool are the global pooling and maximum pooling layers, respectively, and Knernel represents the convolution kernel operation of 1x 1; sigma is an activation function Sigmoid;
the core of the ResNet is a residual mapping module which has autocorrelation correction capability and can adjust errors of a network bottom layer, and generally, expanding the network depth is helpful for enhancing the stability of a model and improving the network performance, but in some experiments, a newly added network layer cannot always enable the model to obtain a better effect when fitting the identity mapping. Assuming that the desired mapping to be learned in the local output nodes of the neural network is F (x), studies have shown that setting the desired mapping of the output nodes of the network as a residual mapping makes it easier to train the network in actual operation, as shown in fig. 13a-13 b.
For feature information X received by the network layer, the residual network layer changes its output form in the forward (forward) computation to F (X) -X, even if residual = ideal map-input, as follows:
S t+1 =σ(S t +R(S t ,K))
wherein: st is the feature information transmitted by the previous network layer, st +1 is residual output, K is two-dimensional convolution kernel operation, the coefficient R defaults to 1, and sigma is an activation function ReLu.
The special mechanism of the residual error network enables error learning of a network layer to be fundamentally distinguished from a traditional network structure, so that the error generated by a network bottom layer is more sensitive, the structural design of a convolution residual error network is referred to in the text, the residual error network is used for correcting the error after a channel attention module, the model structure is deepened in a mode of properly increasing the number of channels, high-order characteristics of an inventory target are gradually abstracted and combined, and the character image recognition of the shelf label is realized by matching with a time sequence module.
4. Checking statistical process of system
The checking management window is used for automatically checking commodities on shelves in a certain or all camera monitoring pictures, realizing business logic based on heavy-load target detection (SSDmobilent) and a shelf label plate identification model (ICRNN) together, and finally realizing quantity and category statistics of the commodities in the pictures, matching of the commodities with the shelves and character statistical information display and the like, wherein the checking process is shown in figure 2.
The specific checking flow comprises the following steps:
firstly, reading an image list according to a selected camera, and acquiring commodity information contained in a picture according to the category and coordinate information output by a target detection model;
next, text recognition is performed on the detected tag area to obtain a unique identification number of the shelf, such as a character A2 in fig. 4.
And finally, performing attribution matching according to the coordinate position of the label plate of the commodity shelf and the detected commodity, wherein the attribution matching means that two label plates are arranged on two sides of each shelf, the coordinate position of the label plate on the image is taken as a reference, the center coordinate of the commodity is detected to be positioned between the two same label plates, the commodity is matched with the shelf corresponding to the label plate, and the inventory result is displayed on a GUI system interface after matching and is synchronously updated with the database.
The inventory information table in this database is shown in table 1.
TABLE 1 inventory results parameter Table
5. Positioning retrieval process of system
The retrieval window is used for inquiring the quantity and the position of the stored goods on site, the function is realized based on the recognition result of the target detection and label text recognition model, and the implementation process of the retrieval function is shown in fig. 3.
The retrieval window matches the output category, coordinate and character information based on SSD-MobileNet and ICRNN models in a corresponding table of a database through retrieval key words, after a retrieval object is selected, the system automatically filters identification information of irrelevant commodities through a set main key, the shelf label number of the target commodity and the quantity of the target commodity are output to an inventory system user interface in a text mode, and finally visualization of retrieval results is achieved as shown in figures 4-10.
The visual model combined with the target detection and image text recognition network, which is constructed by the invention, realizes non-contact remote checking operation, the system realizes data transmission tcp protocol through the Python and MySQL databases in the aspects of statistics, input and maintenance of data information, communicates with the checking system and performs data interaction, checking service data are generated by reasoning and logic checking processes of the image processing model, manual input is not needed, and misjudgment risk of manual operation is avoided. On the aspect of operation performance, the system is trained by combining with a mature target detection model, the identification accuracy of the tag information is obviously improved after the improved ICRNN is adopted, the accuracy rate change curve of the improved ICRNN and the CRNN is shown in figures 14a-14b, and the loss reduction curve of the improved ICRNN and the CRNN is shown in figures 15a-15 b. As can be seen from fig. 14a-14b and fig. 15a-15b, as the training period becomes larger, the loss values of the two networks gradually decrease, and higher accuracy is obtained on the test set gradually. The loss of the ICRNN in the training process and the oscillation condition of the accuracy change curve are superior to those of the CRNN. Meanwhile, the convergence efficiency and the final stable recognition accuracy of CRNN are worse than those of ICRNN.
For the operation and deployment environment of the system, two models based on GPU or CPU operation are provided, and the performance test of business reasoning is carried out on the samples collected in batch at random in the development model, and the result is shown in FIG. 11 and Table 2.
TABLE 2
As can be seen from fig. 11 and table 2, the processing efficiency of the system after the GPU-based accelerated operation is much higher than that of the CPU mode, and compared with manual or inventory-machine type inventory, the proposed system has significant advantages in processing efficiency, and for different batches collected randomly, the processing time of the system in the same processing mode for unit goods, shelves, or sites is relatively stable, which is more favorable for the performance of efficiency evaluation, pre-inventory planning, and inventory layout work compared with the conventional inventory method.
In addition, by combining the type, the coordinate and the text information obtained by the target detection and the image text recognition model, the system can respectively and automatically distinguish, classify and count the warehouse sites containing different commodities and goods shelves according to the proposed processing scheme, so that the fixed deployment relation between the camera and the fixed management area in the partial image processing method is optimized, and the deployment flexibility of the image monitoring equipment in the actual inventory checking application is improved.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (8)
1. A non-contact intelligent checking system is characterized by comprising a basic information module, an image acquisition module, a checking service module and a visual management module;
the basic information module is used for pre-configuring basic information required by system operation;
the image acquisition module is used for acquiring field pictures stored in the storage shelf and preprocessing the acquired images;
the checking service module is used for checking the quantity of the goods on the storage shelf or retrieving the goods according to the retrieval words and storing the obtained checking result or retrieval result into the database;
and the visual management module is used for visually displaying the monitoring pictures and the checking results of the storage shelf.
2. The system of claim 1, wherein the image acquisition module comprises a camera, an image transmission unit and an image preprocessing unit;
the camera and the remote image transmission unit are used for shooting the on-site goods shelf through the monitoring camera on the storage goods shelf and remotely transmitting the shot images to the inventory business module;
and the image preprocessing unit is used for preprocessing the shot image.
3. The system of claim 1, wherein the inventory service module comprises a goods traversal inventory and search unit and a result automatic processing and saving unit;
the goods traversing checking and searching unit is used for checking the number of the goods and the number information of the goods shelf where the goods are located based on the target detection model and the image text recognition network, or retrieving related goods information according to the retrieval key words;
and the result automatic processing and storing unit is used for automatically outputting the checking or searching result to the visual management module and storing the result into the database.
4. The system according to claim 3, wherein the target detection model adopts an SSDMobillent deep learning model for detecting the distribution information of the number, types and coordinates of the objects in the monitoring screen.
5. The system of claim 4, wherein the image text recognition network adopts an ICRNN model, the ICRNN model is improved based on a CRNN model, a channel attention mechanism is introduced into the CRNN model, and a residual network module is used for replacing a traditional CNN layer to recognize characters on label plates of shelves to obtain shelf numbers.
6. The system of claim 5, wherein the basic information module comprises a model loading and parameter definition unit and a database configuration and field management unit;
the database configuration and field management unit is used for defining and inputting the fields of the data forms in the database;
and the model loading and parameter defining unit is used for configuring the target detection model and the hyper-parameters of the image text recognition network.
7. The system of claim 1, wherein the visualization management module comprises a monitoring image preview unit and an AI analysis image presentation unit;
the monitoring image previewing unit is used for previewing the monitoring image shot by the camera;
and the AI analysis image display unit is used for displaying the AI analysis picture into a specified function area through a plurality of function windows, and executing various operations through the operation buttons in the function area.
8. A non-contact intelligent checking method is characterized by comprising the following steps:
step 1: clicking a goods checking button or a goods searching button in the functional window, if the goods checking is selected, turning to the step 2, and if the goods searching is selected, turning to the step 3;
and 2, step: inventory statistics of goods
Step 201: selecting a camera in a storage site, and selecting a shelf image in a picture shot by the camera;
step 202: setting a coordinate interval of an inventory range in the obtained shelf image;
step 203: carrying out target detection in the range of the coordinate interval by using a target detection model, outputting a detection result, and segmenting the detection result to obtain a shelf label plate image;
step 204: inputting the obtained shelf label plate image into an image text recognition network to obtain a character text on the label plate, and obtaining a shelf number according to the character text on the label plate;
step 205: obtaining the shelf number of the commodity shot on the shelf according to the shelf number, repeating the steps 201-204, and finally counting to obtain an inventory result, wherein the inventory result comprises the number and the type of each commodity and a shelf number list;
and step 3: cargo location retrieval
Step 301: searching in the checking result according to the input retrieval key word;
step 302: and outputting the quantity of the commodities and the shelf numbers corresponding to the search keywords.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211023399.6A CN115457458A (en) | 2022-08-25 | 2022-08-25 | Non-contact intelligent checking system and method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211023399.6A CN115457458A (en) | 2022-08-25 | 2022-08-25 | Non-contact intelligent checking system and method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN115457458A true CN115457458A (en) | 2022-12-09 |
Family
ID=84298704
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211023399.6A Pending CN115457458A (en) | 2022-08-25 | 2022-08-25 | Non-contact intelligent checking system and method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115457458A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115529438A (en) * | 2022-01-14 | 2022-12-27 | 深圳进化动力数码科技有限公司 | Goods shelf commodity inspection and checking identification equipment and inspection method |
CN116561167A (en) * | 2023-05-12 | 2023-08-08 | 深圳市正业玖坤信息技术有限公司 | Intelligent factory yield data retrieval system based on image analysis |
-
2022
- 2022-08-25 CN CN202211023399.6A patent/CN115457458A/en active Pending
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115529438A (en) * | 2022-01-14 | 2022-12-27 | 深圳进化动力数码科技有限公司 | Goods shelf commodity inspection and checking identification equipment and inspection method |
CN116561167A (en) * | 2023-05-12 | 2023-08-08 | 深圳市正业玖坤信息技术有限公司 | Intelligent factory yield data retrieval system based on image analysis |
CN116561167B (en) * | 2023-05-12 | 2024-02-27 | 深圳市正业玖坤信息技术有限公司 | Intelligent factory yield data retrieval system based on image analysis |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Santos et al. | Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association | |
CN107463946B (en) | Commodity type detection method combining template matching and deep learning | |
CN101663676B (en) | Method and system for indexing and searching objects of interest across a plurality of video streams | |
CN107463945B (en) | Commodity type identification method based on deep matching network | |
CN111369016B (en) | Image recognition-based on-site operation and maintenance method and system | |
CN115457458A (en) | Non-contact intelligent checking system and method | |
CN110705666A (en) | Artificial intelligence cloud computing display rack goods and label monitoring and goods storage method | |
US20210217017A1 (en) | System and methods for monitoring retail transactions | |
US7136524B1 (en) | Robust perceptual color identification | |
Rahman et al. | Smartphone-based hierarchical crowdsourcing for weed identification | |
CN108345912A (en) | Commodity rapid settlement system based on RGBD information and deep learning | |
CN111061890A (en) | Method for verifying labeling information, method and device for determining category | |
CN113989944B (en) | Operation action recognition method, device and storage medium | |
Hussain et al. | A Simple and Efficient Deep Learning‐Based Framework for Automatic Fruit Recognition | |
CN110032654B (en) | Supermarket commodity entry method and system based on artificial intelligence | |
CN113821674B (en) | Intelligent cargo supervision method and system based on twin neural network | |
CN109670546A (en) | A kind of goods matching and quantity recurrence recognizer based on default template | |
CN115115825B (en) | Method, device, computer equipment and storage medium for detecting object in image | |
Sudana et al. | Mobile application for identification of coffee fruit maturity using digital image processing | |
Defriani et al. | Recognition of regional traditional house in Indonesia using Convolutional Neural Network (CNN) method | |
CN104572930B (en) | Data classification method and device | |
CN113761978A (en) | Method for configuring arithmetic unit for image, electronic device and storage medium thereof | |
Pohudina et al. | Method for identifying and counting objects | |
Allegra et al. | Exploiting egocentric vision on shopping cart for out-of-stock detection in retail environments | |
CN110580299B (en) | Method, system, equipment and storage medium for generating matching diagram of recommended language of object |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |