CN110909743A

CN110909743A - Book checking method and book checking system

Info

Publication number: CN110909743A
Application number: CN201911163170.0A
Authority: CN
Inventors: 章志亮; 曹仁杰; 李钢; 顾明良
Original assignee: BEIJING ZHONGQI ZHIYUAN DIGITAL INFORMATION TECHNOLOGY Co Ltd
Current assignee: BEIJING ZHONGQI ZHIYUAN DIGITAL INFORMATION TECHNOLOGY Co Ltd
Priority date: 2019-11-25
Filing date: 2019-11-25
Publication date: 2020-03-24
Anticipated expiration: 2039-11-25
Also published as: CN110909743B

Abstract

The invention discloses a book checking method and a book checking system. The method comprises the following steps: obtaining a book image of which the single book only contains a single character; respectively inputting each book image into a first pre-trained convolutional neural network one by one for character recognition; sequentially inputting each character recognition result output by the first pre-trained convolutional neural network into a book name library, and determining whether each book obtains a book name; if the book name is obtained, the book name of the book is stored in the checking list; if the book name is not obtained, inputting the image of the book into a second pre-trained convolutional neural network for image recognition; inputting an image recognition result output by a second pre-trained convolutional neural network into a book name library, and determining whether a book obtains a book name; after the book name is obtained, storing the book name into a checking list; otherwise, entering a data area to be marked; and counting book stock according to the book names and pricing of the books stored in the inventory list.

Description

Book checking method and book checking system

Technical Field

The invention relates to a book checking method based on image recognition, and also relates to a book checking system adopting the method, belonging to the technical field of book checking.

Background

For a long time, the inventory work of book physical stores and warehouses is quite complicated, and the practical problems of long time consumption, high input cost of manpower and material resources, influence on business inventory and the like exist. The existing book checking schemes mainly comprise the following two types:

the first book checking scheme is as follows: as shown in fig. 1, a laser bar code scanning gun or RF (radio frequency) device is used to scan one-dimensional bar codes on the book back cover one by one, and then the manual one-by-one processing is performed, and the scanned data is input to a computer system. At present, the book checking scheme is a mode used by most users. The book checking scheme is long in time consumption, high in labor intensity and high in checking cost, and all or part of book entity stores need to stop working to check books.

The second book checking scheme is as follows: an electronic tag is pasted on each book in advance, then the books are put on shelves, and book inventory data is collected by an RFID reader and then input into a computer system during inventory. Although the book checking scheme has greater advantages in processing speed and data acquisition after code pasting than the first book checking scheme, the economic cost is too high, and the book checking scheme is rarely used by users in the book industry. Moreover, due to the problem of the identification rate of the electronic tag, the checking accuracy rate cannot be ensured, and the workload of removing the electronic tag during returning is increased.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a book checking method based on image recognition.

Another technical problem to be solved by the present invention is to provide a book checking system using the above method.

In order to achieve the purpose, the invention adopts the following technical scheme:

according to a first aspect of an embodiment of the present invention, there is provided a book checking method, including the steps of:

step S1: obtaining a book image of which the single book only contains a single character;

step S2: respectively inputting the book images of each single character into a first pre-trained convolutional neural network one by one for character recognition;

step S3: sequentially inputting each character recognition result output by the first pre-trained convolutional neural network into a book name library, and determining whether each book can obtain a book name;

step S4: if a book obtains the title, storing the title of the book into a checking list;

step S5: if a book does not obtain the name of the book, inputting the image of the book into a second pre-trained convolutional neural network for image recognition;

step S6: inputting the image recognition result output by the second pre-trained convolutional neural network into a book name library, and determining whether a book with a book name which is not obtained obtains the book name;

step S7: if a book name is obtained from a book in the step S6, the book name and the pricing are stored in a checking list; otherwise, entering a data area to be marked;

and step S8, counting book stock according to the book names and pricing of the books stored in the inventory list.

Preferably, the book spine image of a certain bookshelf is acquired for preprocessing before the book image of a single book containing only a single character is acquired.

Preferably, the pre-processing of the collected book spine image comprises the following sub-steps:

step 10: performing image segmentation on the book spine image to obtain a single book image;

step 11: and performing font segmentation on the single book image to obtain a book image of which the single book only contains a single character.

Preferably, the process of obtaining the single book image comprises the following sub-steps:

step S100: obtaining a valid spine image from the book spine image;

step 101: and segmenting the effective book spine image to obtain a single book image.

step S1010: converting the effective spine image into a grayscale image;

step S1011: filtering the gray level image;

step S1012: extracting an edge point set of the single book image from the gray level image;

step S1013: and performing linearization processing on the edge point set of the single book image to obtain an effective edge line of the single book image so as to cut out the single book image.

Preferably, the first pre-trained convolutional neural network is trained by the following sub-steps:

step S20: establishing the first convolutional neural network;

step S21: and training the first convolution neural network to obtain first convolution neural network model data.

Preferably, the second pre-trained convolutional neural network is trained by the following sub-steps:

step S60: establishing the second convolutional neural network;

step S61: and training the second convolutional neural network to obtain second convolutional neural network model data.

Preferably, the first convolutional neural network and the second convolutional neural network respectively comprise an input layer, N convolutional layers and pooling layers which are alternately arranged, a full-connection layer and an output layer in sequence, and N is a positive integer;

and obtaining corresponding convolutional neural network model data by setting the filtering parameter of each convolutional layer and the pooling length of each pooling layer.

According to a second aspect of the embodiments of the present invention, there is provided a book checking system, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the following steps when executing the program:

According to a third aspect of embodiments of the present invention, there is provided a machine readable medium having stored thereon a computer program which when executed by a processor performs the steps of:

The book checking method provided by the invention is characterized in that the collected book placing images on a certain bookshelf are preprocessed to form images of single books, and then the images of the single books are subjected to character segmentation to form single characters; then, recognizing characters and images of a single book by using a corresponding convolutional neural network; and then the related technology of the database is used for keeping the title information of the character recognition at an accurate application level. By the method, the problem of application pain in bookstore inventory and shelf management can be effectively solved, inventory efficiency is greatly improved, and inventory cost is reduced.

Drawings

FIG. 1 is a schematic diagram of an apparatus for checking books in the prior art;

FIG. 2 is a schematic diagram of a spine image of a book collected in the book checking method provided by the present invention;

FIG. 3 is a schematic diagram of an effective spine image cut from collected spine images in the book inventory method provided by the present invention;

FIG. 4 is a schematic diagram of a single book image obtained by segmenting an effective spine image in the book checking method provided by the present invention;

FIG. 5 is a schematic flow chart of a book checking method according to the present invention;

fig. 6 is a schematic structural diagram of a book checking system provided by the present invention.

Detailed Description

The technical contents of the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments.

The book checking method provided by the invention is a method for realizing fast book checking based on an artificial intelligence visual recognition technology, and is used for realizing fast checking of books placed on a bookshelf and a goods space in a physical bookstore store and a book warehouse. As shown in fig. 2, the book inventory method includes the following sub-steps:

step S1: a book image is obtained in which the sheet contains only a single character.

And acquiring a book spine image of a certain bookshelf for preprocessing before acquiring a book image of which the single book only contains a single character.

As shown in fig. 3, a book spine displayed on a certain bookshelf is photographed by a mobile phone, a camera, a high-definition camera or other special image acquisition equipment to acquire a spine image of a book, and the acquired spine image of the book is preprocessed to obtain a book image of which a single book only contains a single character; the method comprises the following steps of acquiring a book spine image of a book, wherein the book spine image of the book is preprocessed, and the method comprises the following substeps:

step S10: and carrying out image segmentation on the collected spine image of the book to obtain a single book image.

Because the collected spine images of the books comprise all the books displayed on a certain shelf, if the number of each kind of books needs to be counted, a single book image needs to be obtained from the collected spine images of the books; obtaining the single book image comprises the following substeps:

step S100: a valid spine image is obtained from the captured spine image of the book.

Due to the randomness of data collection, not all the collected spine images of books are valid data, so that the valid images included in the collected spine images of books need to be cut out.

The process of obtaining the effective image from the collected spine image of the book is as follows: copying the collected spine images of the books to obtain two identical spine images of the books; converting the spine image of one book into a gray image by using a color space conversion function, wherein in the gray image, the gray value of the book part is continuously changed due to the fact that the background has the same gray value, namely the gray change rate of the background is relatively stable, and the book part has characters or small color pictures, namely the gray value change rate of the book part is relatively large; then, transversely scanning the book spine gray image of the book, extracting vertex coordinates of the effective image from the book spine gray image of the book according to a boundary between the gray change rate of the background and the gray change rate of the book part, and obtaining coordinates (upper left, lower left, upper right and lower right) of four vertexes of the effective image; as shown in fig. 4, the effective spine image is cut out from the spine image of another book based on the vertex coordinates of the effective image.

Step 101: and segmenting the effective spine image to obtain a single book image.

And obtaining a single book image according to the effective spine image obtained in the step S100, and comprises the following substeps.

Step S1010: the effective spine image is converted to a grayscale image.

The effective spine image is converted to a grayscale image using a color space conversion function.

Step S1011: and carrying out filtering processing on the gray level image.

And (3) converting the effective spine image into a gray image, and realizing the enhancement of points in the y-axis direction of each image area according to the following two-dimensional Gaussian function.

Wherein e represents an exponential function with a natural constant e as a base, σ represents a standard deviation, x represents a value on an x axis of the whole effective spine image, and y represents a value on a y axis of the whole effective spine image; the standard deviation of the two-dimensional Gaussian function is utilized to have a standard deviation in the x direction and the Y direction respectively, different values of all pixels of each divided image area in the x direction and the Y direction are set according to the gray value enhancement degree of the gray image of the effective spine image actually needed, so that the pixels of each divided image area in the x direction or the Y direction are correspondingly increased or decreased, and the effect of the effective spine image in the Y direction is enhanced.

Step S1012: and extracting the edge point set of the single book image from the gray level image after the filtering processing.

And extracting the edge point set of the single book image by setting the ranges of the high value and the low value of the edge extraction function so as to highlight the local edge in the single book image.

Step S1013: and (4) performing linearization processing on the edge point set of the single book image to obtain an effective edge line of the single book image so as to cut out the single book image.

Due to the existence of noise and blur, the boundary formed by the collection of edge points of the extracted single book image may become wide or may be discontinuous at a certain point. Therefore, it is necessary to remove some edge points or fill up edge discontinuities from the edge point set of the single book image by using a linear algorithm, and connect these edge points into a complete line to obtain the effective edge line of the single book image. Specifically, the process of linearizing the edge point set of the book image is as follows: establishing a rectangular coordinate system, gathering edge points of the single book image, starting from a first edge point, bringing the coordinates of each edge point into the rectangular coordinate system one by one to obtain a slope, removing the edge points outside the line where the slope is located, and filling edge discontinuity points to obtain the effective edge line of the single book image. As shown in FIG. 5, the individual book images are cut out according to their effective edge lines.

Step 11: and performing font segmentation on the single book image to obtain the book image of which the single book only contains a single character.

Due to the characteristics of different design styles and different thicknesses of books, the single book image obtained in the step S10 needs to be preprocessed and then subjected to font segmentation to obtain a book image of which the single book only contains a single character.

Specifically speaking, if the book in the single book image is a book, in order to avoid influencing the recognition of the fonts due to the thinner thickness of the book spine of the book, all pixels in the single book image need to be transversely amplified, so that the whole single book image can be amplified. If the fonts in the single book image are double-row fonts and the fonts can be separated or tightly aligned, extracting the coordinates of each font in the single book image to generate an intermediate file, wherein the intermediate file comprises the single-row fonts and the double-row fonts, and cutting the double-row fonts into the single-row fonts according to the principle that the pixel values between the left-row fonts and the right-row fonts are not changed. And finding out the starting and stopping positions of each font by adopting a horizontal projection method for the single-column fonts in the obtained single book image so as to conveniently segment the book image of which the single book only contains a single character according to the starting and stopping positions of each font.

Step S2: and respectively inputting the book images of each single character into a first pre-trained convolutional neural network one by one for character recognition.

The book image of each single character is respectively input into the first pre-trained convolutional neural network, so that the characters with the highest probability of appearing in the word stock data set in the book image of each single character can be identified, namely the characters related to the book name appear in the word stock data set. Since the name, author, and publishing company of each book are generally printed on the spine of the book, the name of the book referred to in the present invention mainly refers to the name, author, and publishing company of each book. The characters related to the name, the author and the publishing company printed on the spine of each book can be identified by the first pre-trained convolutional neural network, wherein the characters are the characters with the highest probability in the character library data set.

The forming process of the word stock data set comprises the following steps: generating a label file related to coding according to the content of the existing book name file (existing characters in the existing book name library), and obtaining a library file; in the label file, each character corresponds to one code, corresponding characters are found out from a character library according to the codes in the label file, the characters are converted into character images to obtain a character library image file, and images corresponding to the characters in the character library image file are rotated, noisy and the like to generate a character library data set. For example, taking a character as an example, the character has multiple fonts, each font corresponds to a character image, that is, each character has multiple character images, and each character image of the character is respectively rotated, noised, and the like to generate a data set of the character. Therefore, the generated word stock data set is a set of character images of characters existing in the existing title library after being processed.

The first pre-trained convolutional neural network is trained by the following sub-steps:

step S20: a first convolutional neural network is established.

In the step, the established first convolutional neural network sequentially comprises an input layer, N convolutional layers and pooling layers which are alternately arranged, a full-connection layer and an output layer, wherein N is a positive integer. Wherein the number of convolutional layers and pooling layers is set according to the required neural network model data. For example, as shown in the figure, as a specific embodiment of the present invention, the first convolutional neural network provided in this embodiment includes 2 convolutional layers and pooling layers alternately arranged, that is, the first convolutional neural network sequentially includes an input layer, a convolutional layer, a pooling layer, a fully-connected layer, and an output layer.

Step S21: and training the established first convolution neural network to obtain first convolution neural network model data.

And respectively inputting the training data set and the test data set into the established first convolution neural network, and obtaining the required first convolution neural network model data through multiple times of training. In the process of training the first convolutional neural network, the filtering parameters of each convolutional layer are set, and the pooling length of each pooling layer is set, so that ideal first convolutional neural network model data can be obtained. The training data set and the test data are obtained from a word stock data set, and specifically, the proportion of the training data set and the test data is selected according to the size of an obtained word stock image file. For example, 80% of the word stock data set is used as the training data set, and 20% of the word stock data set is used as the testing data set.

Step S3: and sequentially inputting each character recognition result output by the first pre-trained convolutional neural network into a book name library, and determining whether each book obtains a book name.

The character recognition result output by the first pre-trained convolutional neural network means that the character with the highest probability of appearing in the word stock data set by the name, the author and the publishing company printed on the spine of each book is recognized by the first pre-trained convolutional neural network. And the first character of each identified book is brought into the book name library to find out all book names containing the characters, and the second character of each identified book is brought into the book name containing the last character found out by the last character according to the next character, so that the range of the found book names is further narrowed, and the book names of each book are finally obtained. The obtained book name of each book comprises the name, author and publishing company of the book. And finding out the pricing of the book from the book name library according to the obtained book name of each book so as to facilitate the subsequent inventory of the books.

Step S4: and if a book obtains the book name, storing the book name of the book into the checking list.

If the book name of a book can be obtained according to the method of step S3, the book name of the book and the pricing found in the book name library according to the book name are respectively saved in the inventory table.

if the name of a book is not obtained according to the method of step S3, the image of the book is input into a second pre-trained convolutional neural network for image recognition.

Step S6: and inputting the image recognition result output by the second pre-trained convolutional neural network into a book name library, and determining whether a certain book without the book name obtains the book name.

And respectively inputting the single book images obtained in the step S10 into a second pre-trained convolutional neural network to confirm whether the single book images can be found from the image data set, and further confirm the book name and pricing information of each book from the image data set.

Wherein the formation process of the image data set comprises the following steps: and respectively labeling the single book images obtained in the step S10, wherein the labeled contents are the book number and the book name of each book. And all the marked book images form an image data set.

The second pre-trained convolutional neural network is trained by the following sub-steps:

step S60: a second convolutional neural network is established.

In the step, the established second convolutional neural network sequentially comprises an input layer, N convolutional layers and pooling layers which are alternately arranged, a full-connection layer and an output layer, wherein N is a positive integer. Wherein the number of convolutional layers and pooling layers is set according to the required neural network model data.

Step S61: and training the established second convolutional neural network to obtain second convolutional neural network model data.

And respectively inputting the training image data set and the test image data set into the established second convolutional neural network, and obtaining the required second convolutional neural network model data through multiple times of training. In the process of training the second convolutional neural network, the filtering parameters of each convolutional layer are set, and the pooling length of each pooling layer is set, so that ideal second convolutional neural network model data can be obtained. The training image data set and the testing image data are obtained from the image data set, and specifically, the proportion of the training image data set and the testing image data is selected according to the size of the obtained word stock image file. For example, 70% of the image data set is used as the training image data set, and 30% of the image data set is used as the test image data set.

Step S7: if the book name is available in the book in the step S6, the book name and the pricing are stored in the checking list; otherwise, entering a data area to be marked.

If the book name of a book can be obtained according to the method of step S6, the book name of the book and the pricing found in the book name library according to the book name are respectively saved in the inventory table.

If a book is subjected to image comparison by adopting a second convolutional neural network, the name of the book still cannot be obtained, the book can be placed into the data area to be marked, and the image of the book in the data area to be marked is respectively marked in an artificial mode and then is merged into an image data set, so that the name of the book can be obtained according to the latest image data set when the same book image is encountered next time.

The quantity of each kind of books can be counted according to the book names of the books stored in the inventory list and the pricing of each kind of book, and the inventory of the books can be obtained according to the pricing of each kind of book. Wherein each book refers to the name of the book, the author, and the publisher being the same.

As shown in fig. 6, an embodiment of the present invention further provides a book checking system, including: a memory 61 and a processor 62, wherein the memory 61 stores a computer control program, and when the computer control program is executed by the processor 62, the steps of the book checking method provided by the present invention are realized (the steps S1-8 described above). In addition, the embodiment of the present invention further provides a machine-readable medium, which stores a computer control program, and the computer control program, when being executed by a processor, implements the steps (such as the steps S1-S8) of the book checking method provided by the present invention.

It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on machine-readable media (e.g., computer-readable media), which can include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.

The book checking method and the book checking system provided by the invention are explained in detail above. It will be apparent to those skilled in the art that various modifications can be made without departing from the spirit of the invention.

Claims

1. A book checking method is characterized by comprising the following steps:

2. The book inventory method of claim 1, characterized in that:

3. The book inventory method of claim 2, characterized in that:

the method comprises the following steps of preprocessing the collected book spine image, wherein the method comprises the following substeps:

4. The book inventory method of claim 3, characterized in that the process of obtaining the single book image comprises the substeps of:

step S100: obtaining a valid spine image from the book spine image;

5. The book inventory method of claim 4, characterized in that the process of obtaining the single book image comprises the substeps of:

step S1010: converting the effective spine image into a grayscale image;

step S1011: filtering the gray level image;

6. The book inventory method of claim 1, characterized in that said first pre-trained convolutional neural network is trained by the following sub-steps:

step S20: establishing the first convolutional neural network;

7. The book inventory method of claim 1, characterized in that said second pre-trained convolutional neural network is trained by the following sub-steps:

step S60: establishing the second convolutional neural network;

8. The book inventory method of claim 1, characterized in that:

the first convolutional neural network and the second convolutional neural network sequentially comprise an input layer, N convolutional layers and pooling layers which are alternately arranged, a full-connection layer and an output layer, and N is a positive integer;

9. A book inventory system comprising a memory, a processor, and a computer program stored on said memory and executable on said processor, characterized in that: the processor implements the following steps when executing the program:

10. A machine readable medium having a computer program stored thereon, wherein the program when executed by a processor implements the steps of: