CN112528722A - Book checking method and system based on SSD-ResNet algorithm - Google Patents

Abstract

The invention relates to a book checking method and a book checking system based on an SSD-ResNet algorithm, which comprises a first step; the method comprises the steps that an image acquisition device acquires bookshelf book images; step two: transmitting the bookshelf book images to an image processing unit; step three: the image processing unit processes the collected bookshelf book images through an SSD-ResNet algorithm until the bookshelf book images meet image recognition; step four: restoring the collected image information into a digital bookshelf; step five: and transmitting the digital bookshelf to image acquisition equipment for visual checking and arranging the bookshelf. The invention solves the technical problem that a great deal of manpower and time are often consumed for book checking.

Description

Book checking method and system based on SSD-ResNet algorithm

Technical Field

The invention belongs to the field of artificial intelligence, and particularly relates to a book checking method and system based on an SSD-ResNet algorithm.

Background

The circulation of books, move and extensive open a shelf borrow the random of placing of reader to books under the mode, all probably cause the actual information of books to be inconsistent with the library information, for example can often appear the reader and find certain book in library bibliography retrieval system, but nevertheless can't find the condition of this book to corresponding the shelf position, influenced borrowing of books.

The book checking is used as a basic normalization work of the library, and is helpful for a librarian to know the condition of paper resources of the library. Through checking, the staff can accurately master the number, the positions and the shelf conditions of the books, and the service level and the management quality of the library are improved; the inventorying often requires a great deal of manpower and time.

In the prior art, the robot is used for book checking, the robot has the advantages of unattended operation, non-contact, high efficiency and the like, can break through the bottleneck problems of the traditional book checking mode, such as manpower, attention, patience, time, equipment, cost and the like, can radically improve book checking work, and also enables conventional book checking to be possible. However, the robot has some problems that the robot cannot be widely popularized and applied for a long time in the future, and the accuracy of information reading of the robot, a positioning algorithm, an error correction capability and human interaction have an optimized space.

Aiming at the technical problem, the invention provides a book checking method and system based on an SSD-ResNet algorithm, and the book checking method and system are generated by the scheme.

Disclosure of Invention

The invention aims to provide a book checking method based on an SSD-ResNet algorithm, which aims to solve the technical problem that a large amount of labor and time are consumed for book checking;

in order to achieve the purpose, the invention specifically provides the following technical scheme: a book checking method based on an SSD-ResNet algorithm comprises the following steps of; the method comprises the steps that an image acquisition device acquires bookshelf book images; step two: transmitting the bookshelf book images to an image processing unit; step three: the image processing unit processes the collected bookshelf book images through an SSD-ResNet algorithm until the bookshelf book images meet image recognition; step four: restoring the collected image information into a digital bookshelf; step five: and transmitting the digital bookshelf to image acquisition equipment for visual checking and arranging the bookshelf.

Further, book labels corresponding to the SSD-ResNet algorithm are one-dimensional codes.

Further, the SSD-ResNet algorithm comprises the steps of: firstly, extracting a characteristic diagram matrix from an image through a ResNet network layer; secondly, the following steps: generating feature maps with different scales through convolution operation; thirdly, defining rectangular frames with different sizes in advance for each position in each feature map, and extracting the detection value of the anchor by performing convolution on the feature map by adopting a convolution kernel to extract the detection value of the anchor, including the type and the frame position of the anchor; and fourthly, classifying and regressing all the anchor detection values at the same time.

Further, in step two, the large-scale feature map can be used for detecting small objects, and the small-scale feature map is used for detecting large objects.

Further, in step three, the detected target has c categories, and the SSD needs to predict c +1 confidence values, where the first confidence value refers to the score without target or belonging to the background.

Furthermore, the image acquisition equipment continuously acquires the bookshelf books to form a plurality of bookshelf book images, and the bookshelf book images comprise the labels of the books and the information of the bookshelves and the shelves.

Further, the labels of the books comprise the position, the sequence and the book content information of the books.

Further, the information of the books on the shelf is restored into the digital bookshelf by combining the book library and the book shelf, and the book metadata.

And further, counting statistics is carried out through information comparison of the visual digital bookshelf and the books on the bookshelf, and bookshelf arrangement is completed.

The invention also aims to provide a book checking system based on the SSD-ResNet algorithm, so as to solve the problem that in the prior art, the robot has an optimized space in the aspects of information reading accuracy, positioning algorithm, error correction capability and human interaction.

In order to achieve the purpose, the invention specifically provides the following technical scheme: a book checking system based on SSD-ResNet algorithm comprises

The image acquisition unit is used for acquiring, storing and transmitting bookshelf images;

the image processing unit is used for processing the images, detecting the images and identifying book labels;

and the data application unit is used for arranging the book label information, reducing the book label information into a digital bookshelf by combining the original image, and performing bookshelf visual management and statistics.

Compared with the prior art, the invention has the advantages that:

firstly, the implementation cost is low, and librarians can take pictures of the bookshelf layer by using a mobile phone camera in an application program. And the RFID checking equipment is expensive, and the purchase quantity in a hall is limited. Also, the cost of one-dimensional/two-dimensional barcode tags has an absolute advantage compared to RFID tags.

The implementation process is simple, the RFID is used for book checking management, the book is required to be processed in a centralized mode through the RFID tag, and meanwhile, various RFID devices are required to be debugged: an inventory car, a handheld inventory device, and the like. And the checking technology of the deep neural network is adopted, and only one-dimensional or two-dimensional bar codes are needed to be pasted on the book back.

And operation and maintenance are simple, compared with maintenance of software and hardware systems such as an RFID (radio frequency identification) checking vehicle and handheld equipment, the checking technology based on the deep neural network only needs a software-shaped checking system and a bar code, and the maintenance difficulty is greatly reduced.

And fourthly, the service threshold is low, the RFID checking technology needs to train the librarian for a long time, and meanwhile, the change of the service flow is needed to be realized. And the book inventory technology based on the deep neural network only needs a librarian to have basic photographing capability, can photograph relatively clear shelf pictures, and can complete the inventory of shelf books by a server.

And fifthly, the order of books on the shelf can be identified, the RFID checking technology is influenced by the instability of radio frequency signals, the order of the books in the shelf can not be accurately acquired, and the books on the shelf can be accurately sequenced through the position of the one-dimensional/two-dimensional bar code in the photo by the book checking technology based on the deep neural network.

And sixthly, the inventory accuracy is high, and based on experiments and data analysis, the inventory accuracy rate of the book based on the deep neural network is verified to be more than 99% compared with that of the RFID inventory, and the average time of each book is less than 1 second. Due to the divergence of radio frequency signals, the reading and writing equipment can often misread books at other layers, and the RFID tag is easily shielded and reflected by a steel bookshelf, so that the checking result is influenced.

Seventh, the checking speed is high, the book checking speed based on the deep neural network is smaller than 1 second per book on average, and the checking speed is high when the number of the bars in the same photo is more. Compared with the RFID inventory technology, the technology has obvious speed advantage.

And eighthly, the applicability is strong, and the RFID checking technology can only finish checking books which cannot be normally placed such as special-shaped books and oversized books by means of image acquisition, image recognition and machine learning technologies. And the book checking technology based on the deep neural network can finish checking of books such as special-shaped books, oversized books and the like only through photos.

Ninth, the system has better compatibility with other systems, can be compatible with a magnetic stripe anti-theft system of a library, and realizes borrowing, checking and theft prevention of books; the system is also compatible with an RFID anti-theft system, and book borrowing, inventory and theft prevention are realized; but also can be docked with other automatic systems (such as robot inventory) only by providing shelf pictures by the equipment.

And ten, the checking site can be restored, the checking technology based on the deep neural network can store the photo of the shelf for a long time, and if the situation that the book is lost is met, the librarian calls the photo to complete the checking of the book. In the RFID checking technology, the RFID signal acquisition is disposable, and the checking site cannot be restored, so that the checking workload is greatly increased.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

FIG. 1 is a schematic diagram of a system according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of image acquisition according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an SSD-ResNet algorithm according to an embodiment of the present invention;

FIG. 5 is a comparison graph of the inventory accuracy of RFID, one-dimensional bar code, and two-dimensional bar code in accordance with an embodiment of the present invention;

fig. 6 is a comparison chart of the inventory recall ratio of the RFID, the one-dimensional barcode, and the two-dimensional barcode according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

As shown in FIG. 1, a book inventory system based on SSD-ResNet algorithm comprises

S101, an image acquisition unit for acquiring, storing and transmitting bookshelf images;

s102, an image processing unit is used for processing images, detecting the images and identifying book labels through an SSD-ResNet algorithm;

and S103, a data application unit for arranging the book label information, reducing the book label information into a digital bookshelf by combining the original image, and performing bookshelf visual management and statistics.

The image processing unit and the data application unit are generally used in a computer, but are not limited to a computer.

As shown in FIG. 2, a book inventory method based on SSD-ResNet algorithm

S201, collecting images, continuously collecting the bookshelf photos by using image collecting equipment, and recording the sequence of the photos, the bookshelf where the photos are located, the shelves and other information.

In the image acquisition process, the conditions of on-site light, book discharge, book contamination, interferents and the like need to be comprehensively considered, so that the acquired images are clear, and book labels are omitted.

S202, storing the image, storing the acquired image file, and paying attention to the performance, capacity, service life, economy and the like of the equipment in the storage process.

S203, transmitting the image, and transmitting the image file to the image processing unit by using a proper data transmission means when the condition allows.

S204, processing the image, and processing the received photo to enable the photo to meet the requirement of image recognition.

S205, detecting and recognizing the image, and carrying out label recognition on the processed photo through a recognition deep learning algorithm. The deep learning algorithm is designed and used with a flexible algorithm engine, and different algorithm kernels can be replaced in the algorithm engine according to different book labels, such as SSD, Fast R-CNN, ResNet, Mask R-CNN, rfcn-rcnn, SSD-mobile and the like. By identifying the book tags, the location, order and content of the book tags may be further determined.

S206, restoring the bookshelf, and restoring the collected information into a digital bookshelf by combining the book library, the book shelf information, the book metadata and the like.

S207, counting, namely comparing the digital bookshelf images and the bookshelf images with the on-shelf information of the books, and realizing analysis and visual application of visual angles such as misplaced, missed, insufficient, full and stained books.

As shown in fig. 4, the SSD (Single Shot multi box Detection) model is mainly formed by connecting a basic network block and a plurality of multi-scale feature blocks in series. The basic Network block is used to extract feature map from the original picture, and the algorithm using ResNet (Residual Network) as the basic Network is SSD-ResNet.

The Faster R-CNN algorithm is specifically as follows:

extracting a characteristic diagram matrix from the image through a ResNet network layer, and then generating a series of characteristic diagrams with different scales through a series of convolution operations (or Powing), wherein the large-scale characteristic diagram (the characteristic diagram at the front) can be used for detecting small objects, and the small-scale characteristic diagram (the characteristic diagram at the back) is used for detecting large objects; then, for each position in each feature map, a rectangular frame (namely anchor) with different sizes is predefined, and the feature map is convolved by a convolution kernel to extract the detection value of the anchor, wherein the method comprises the following steps: the type of anchor (note that SSD treats the background as a special type, if the detected objects have c types, SSD actually needs to predict c +1 confidence values, where the first confidence refers to the score of no object or belonging to the background), the border position of anchor; and finally, simultaneously classifying and regressing all the anchor detection values.

The ResNet solves the problem that the performance of the neural network is not reduced along with the deepening of the network layer number. The core idea is to introduce identity mapping, allow the original input information to be directly transmitted to the following layers, and the model is degraded into a shallow network.

Fast inventory framework based on deep neural network: aiming at the defects of the existing inventory method in the industry, a book inventory frame based on a deep neural network is provided by adopting a target detection algorithm. The frame uses one-dimensional/two-dimensional bar codes as labels, the labels are pasted on the backs of the book spines, a mobile phone is used for photographing or recording small videos for books on shelves, then a plurality of one-dimensional/two-dimensional bar codes in the photos (or video frames) are identified through a target detection algorithm, and finally reading of book codes is achieved through bar code identification. The one-dimensional bar code can adopt EAN code, 39 code, 128 code, Codabar (Codabar code) and the like, the two-dimensional bar code can adopt Data Matrix code, and the one-dimensional bar code/the two-dimensional bar code can meet the inventory requirement of the thickness of a centimeter-level book.

On the basis of a one-dimensional/two-dimensional bar code label, a book inventory frame based on a deep neural network completes inventory of books on a shelf in three stages:

a detection stage: the detection of the area where the target (bar code) is located is completed through a target detection algorithm;

a sequencing stage: intercepting a detection target (bar code) and sequencing according to a target (bar code) coordinate;

and (3) identification: reading the target (bar code) value and calculating the checking result.

In the three phases, the most core detection phase adopts a fast R-CNN model or an SSD-ResNet model which is common in the target detection algorithm. The former is a two-stage method, is used for detecting two-dimensional bar codes, and has the main idea that a series of sparse candidate frames are generated through a CNN (Convolutional Neural network), and then the candidate frames are classified and regressed, so that the method has the advantage of high accuracy; the latter is a one-stage method, is used for the detection of the one-dimensional bar code, the thought is to carry on the intensive sampling in different positions of the picture evenly, can adopt different yardstick and length-width ratio while sampling, then classify and regress directly after utilizing CNN to extract the characteristic, the whole process only needs one step, the advantage is fast, the resolution ratio of the picture is lower, can guarantee the precision detected too.

The prior art and the inventorying speed and advantages of the embodiments of the present invention are discussed next:

experimental environment and procedure: checking three types of books through experiments, wherein books with ultrahigh frequency tags commonly used in libraries of colleges and universities are selected through an RFID checking technology; for the two-dimensional bar code, book checking technology based on fast R-CNN is adopted; for the one-dimensional bar code, a book inventory technology based on SSD-ResNet is adopted, and only the two-dimensional bar code is replaced by the one-dimensional bar code.

The test environment consisted of a total of 10 bookshelves 2137 books. In the test, 10 inventory tests are carried out on the 10 bookshelves in sequence, and 100 books are randomly drawn to simulate the borrowed books in each test. In order to simulate a real scene, the books on the bookshelf comprise orderly arranged books and books inclined at a certain angle

In the experiment, in the RFID book inventory process, a librarian uses a handheld RFID inventory device to scan layer by layer to complete inventory; the librarian uses a Hua's P30 mobile phone to take a picture of the book shelf and uploads the picture to the prototype system based on the inventory of the deep neural network.

The experimental results are as follows: in order to evaluate the experimental effect, a book inventory mode based on a deep neural network and a book inventory mode based on RFID are subjected to a comparison test experiment, and comparison indexes are four aspects of inventory speed, accuracy, precision and recall rate of a system.

Counting speed: the results of the RFID inventory and the inventory experiment based on the deep neural network are shown in Table 1. In 10 experiments, the average time of RFID inventory is 1109.2 seconds, the average time of two-dimensional barcode inventory is 720.6 seconds, and the average time of one-dimensional barcode inventory is 723.5 seconds. As can be seen from table 1, the inventory technology based on the deep neural network is significantly superior to the RFID inventory technology in speed, which is about 65% of the RFID inventory, i.e., the inventory technology based on the deep neural network is improved in speed by about 35%.

TABLE 1 speed comparison of RFID, one-dimensional barcode, two-dimensional barcode inventory

Counting performance: and evaluating the checking performance by adopting three indexes of accuracy, precision and recall rate. First, 4 statistical attributes of the sample are introduced:

TP: the True Positive, namely the checking system judges correctly, and the book is on the shelf;

TN: true Negative, i.e. the checking system judges correctly, this book is not on the shelf;

FP: false Positive, i.e. the checking system judges wrongly, this book is actually not on shelf, because probably the RFID reading head reads through the shelf;

FN: false Negative, i.e. the checking system judges wrongly, this book is actually on the shelf, because the RFID label is missed, the one-dimensional/two-dimensional bar code label is not recognized.

TABLE 2 comparison of RFID, one-dimensional barcode, two-dimensional barcode inventory performance

The test data set was tested to obtain statistical attribute values for each inventory as shown in table 2. According to the 4 statistical attributes of the table 2, the accuracy (a), the accuracy (P) and the recall rate (R) are adopted for comprehensive evaluation of the checking result. The evaluation index formula is defined as follows:

the accuracy, precision and recall of the three types of book inventory can be obtained according to the table 2 and the formula 1. In the aspect of accuracy, the RFID checking accuracy is about 94.1%, and the accuracy of checking the one-dimensional bar code and the two-dimensional bar code is 100.0%. That is to say, book inventory technology based on the deep neural network can identify all books on shelf more correctly, and compared with RFID book inventory, the book inventory technology improves by nearly 6 percentage points.

In terms of accuracy and recall, fig. 5 and 6 show the comparison between the accuracy and recall of the three types of book inventory according to the results in table 2. In the aspect of inventory accuracy, the RFID book inventory is about 90.7%, the one-dimensional bar code based on the deep neural network is about 99.9%, and the two-dimensional bar code based on the deep neural network is about 99.3%. In the aspect of the inventory recall rate, the RFID book inventory is about 95.8%, the one-dimensional bar code based on the deep neural network is about 99.9%, and the two-dimensional bar code based on the deep neural network is about 99.2%. Namely, the accuracy of the inventory technology based on the deep neural network can reach more than 99%, and is greatly improved compared with the RFID inventory technology.

Claims (10)

1. A book checking method based on an SSD-ResNet algorithm is characterized in that: comprises the steps of A; the method comprises the steps that an image acquisition device acquires bookshelf book images; step two: transmitting the bookshelf book images to an image processing unit; step three: the image processing unit processes the collected bookshelf book images through an SSD-ResNet algorithm until the bookshelf book images meet image recognition; step four: restoring the collected image information into a digital bookshelf; step five: and transmitting the digital bookshelf to image acquisition equipment for visual checking and arranging the bookshelf.

2. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein: and the book label corresponding to the SSD-ResNet algorithm is a one-dimensional code.

3. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein the SSD-ResNet algorithm comprises the following steps: firstly, extracting a characteristic diagram matrix from an image through a ResNet network layer; secondly, the following steps: generating feature maps with different scales through convolution operation; thirdly, defining rectangular frames with different sizes in advance for each position in each feature map, and extracting the detection value of the anchor by performing convolution on the feature map by adopting a convolution kernel to extract the detection value of the anchor, including the type and the frame position of the anchor; and fourthly, classifying and regressing all the anchor detection values at the same time.

4. The book inventory method based on SSD-ResNet algorithm of claim 3, wherein: in step two, the large-scale feature map can be used for detecting small objects, and the small-scale feature map is used for detecting large objects.

5. The book inventory method based on SSD-ResNet algorithm of claim 3, wherein: and step three, c categories are detected, and the SSD needs to predict c +1 confidence values, wherein the first confidence value refers to a score without a target or belonging to a background.

6. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein: the image acquisition equipment continuously acquires the bookshelf books to form a plurality of bookshelf book images, and the bookshelf book images comprise the labels of the books and the information of the bookshelves and the shelves.

7. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein: the labels of the books comprise the positions, the sequence and the content information of the books.

8. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein: the collected information is restored into the digital bookshelf by combining the book library and the book shelf, the information of the books on the shelf and the book metadata.

9. The book inventory method based on SSD-ResNet algorithm of claim 1, wherein: counting statistics is carried out through information comparison of the visual digital bookshelf and the books on the bookshelf, and bookshelf arrangement is completed.

10. A book checking system based on an SSD-ResNet algorithm is characterized in that: comprises that

The image acquisition unit is used for acquiring, storing and transmitting bookshelf images;

the image processing unit is used for processing images, detecting images and identifying book labels through an SSD-ResNet algorithm;

and the data application unit is used for arranging the book label information, reducing the book label information into a digital bookshelf by combining the original image, and performing bookshelf visual management and statistics.

Images