TWI423154B - Book Inventory Management System and Its Method - Google Patents
Book Inventory Management System and Its Method Download PDFInfo
- Publication number
- TWI423154B TWI423154B TW098145906A TW98145906A TWI423154B TW I423154 B TWI423154 B TW I423154B TW 098145906 A TW098145906 A TW 098145906A TW 98145906 A TW98145906 A TW 98145906A TW I423154 B TWI423154 B TW I423154B
- Authority
- TW
- Taiwan
- Prior art keywords
- book
- image
- inventory
- frame
- bookshelf
- Prior art date
Links
Description
本發明是有關於一種庫存管理技術,特別是指一種基於影像(vision-based)的圖書庫存管理系統及其方法。The present invention relates to an inventory management technology, and more particularly to a vision-based book inventory management system and method thereof.
圖書館或書店在開放或營業的過程中,一定會有借出、售出、歸還、進貨、上架,遺失等情形發生,經過一段時間後,圖書數量勢必有所變更;再者,圖書館或書店內的圖書,會因為使用者的翻閱與未正確歸位,造成圖書散落於館(店)內的各角落,圖書放置位置也可能有所改變。所以,為了維護大量的藏書資料,往往需要付出相當高的人事成本;舉例來說,以定期進行人工盤點的方式,來維護目前所有圖書的資料。In the process of opening or opening a library or bookstore, there must be cases of lending, selling, returning, purchasing, loading, losing, etc. After a period of time, the number of books is bound to change; in addition, the library or The books in the bookstore will be scattered in the corners of the hall (shop) due to the user's reading and incorrect homing, and the placement of the book may also change. Therefore, in order to maintain a large amount of library materials, it is often necessary to pay a relatively high personnel cost; for example, to maintain the current data of all books by means of manual inventory.
為了對庫存的圖書進行自動化管理,一種習知的庫存系統,如美國專利US 6,693,539所揭露的「使用無線射頻識別(radio frequency,簡稱RFID)標籤(tag)之物品的庫存系統」,係透過在物品(圖書)上裝設被動式RFID標籤,然後,以RFID讀取器與RFID掃描器建立及更新物品(圖書)的流通狀態,並配合庫存資料庫來儲存及提供圖書的流通狀態以進行追蹤及管理。In order to automate the management of inventories, a conventional inventory system, such as the "inventory system for the use of radio frequency (RFID) tags), as disclosed in U.S. Patent No. 6,693,539, is incorporated herein by reference. A passive RFID tag is installed on the item (book), and then the RFID reader and the RFID scanner are used to establish and update the circulation status of the item (book), and the inventory database is used to store and provide the circulation status of the book for tracking and management.
雖然,利用RFID技術進行圖書管理可以對庫存的圖書進行自動化管理,並降低所需人事成本。但由於無線電信號(radio signal)本身特性,其信號強度容易受到下列因子的影響而造成誤差:室內之障礙物造成無線電信號的吸收與遮蔽(shadowing)效應,以及室內各種電子設備的干擾影響無線電信號的穩定度。而且,由於圖書的販售所得利潤逐年降低,在每本圖書上裝設被動式RFID標籤,無異是提高了每本圖書的成本,且館(店)內尚需配合架設相關硬體設備,並不符合經濟效益。Although the use of RFID technology for book management can automatically manage the inventory of books and reduce the personnel costs required. However, due to the nature of the radio signal, its signal strength is easily affected by the following factors: the absorption and shadowing effects of radio signals caused by obstacles in the room, and the interference of various electronic devices in the room affect the radio signals. Stability. Moreover, as the profit from the sale of books is decreasing year by year, the installation of passive RFID tags on each book is equivalent to increasing the cost of each book, and the related hardware devices need to be set up in the library (shop). Not economically viable.
因此,本發明之目的,即在提供一種圖書庫存管理系統。Accordingly, it is an object of the present invention to provide a book inventory management system.
於是,本發明圖書庫存管理系統,適於與一攝影裝置連接,該系統包含一記憶單元、一影像處理單元,及一庫存資料管理單元。該記憶單元儲存有已建立的至少一庫存資料組,該庫存資料組包括一庫存圖書之一書背影像。該影像處理單元電連接於該記憶單元,該影像處理單元包括一切割模組及一比對模組,該切割模組用以接收該攝影裝置所擷取之一書架的一圖框,並對該圖框進行影像切割,以取得該書架上之圖書的一書背影像,當該系統於一盤點模式時,該比對模組用以於該記憶單元中,進行關於該書架上之圖書的書背影像與該庫存圖書之書背影像的比對。該庫存資料管理單元電連接於該記憶單元及該影像處理單元,當該系統於該盤點模式時,該庫存資料管理單元用以根據該比對模組之比對結果產生該書架上之圖書的一庫存狀態,並將其儲存至該記憶單元。Therefore, the book inventory management system of the present invention is adapted to be connected to a photographing device, the system comprising a memory unit, an image processing unit, and an inventory data management unit. The memory unit stores at least one inventory data set that has been established, and the inventory data group includes a back image of an inventory book. The image processing unit is electrically connected to the memory unit, and the image processing unit includes a cutting module and a matching module, and the cutting module is configured to receive a frame of the bookshelf captured by the camera device, and The frame performs image cutting to obtain a back image of the book on the bookshelf. When the system is in an inventory mode, the comparison module is used in the memory unit to perform a book about the book on the shelf. The comparison between the back image of the book and the back image of the book of the stock book. The inventory data management unit is electrically connected to the memory unit and the image processing unit. When the system is in the inventory mode, the inventory data management unit is configured to generate a book on the shelf according to the comparison result of the comparison module. An inventory status and store it to the memory unit.
本發明之另一目的,即在提供一種圖書庫存管理方法。Another object of the present invention is to provide a book inventory management method.
於是,本發明圖書庫存管理方法,適於藉由與一攝影裝置連接的一系統執行,該方法包含下列步驟。A)接收一圖書之一識別資訊;B)初始化該圖書之一庫存狀態;C)接收該攝影裝置所擷取之一書架的一圖框;D)對該圖框進行影像切割,以取得該書架上之圖書的一書背影像;及E)建立對應該圖書之一庫存資料組,其中,該庫存資料組包括該識別資訊、該庫存狀態,及該書背影像。Thus, the book inventory management method of the present invention is adapted to be executed by a system connected to a photographing apparatus, the method comprising the following steps. A) receiving one of the books to identify the information; B) initializing the inventory status of the book; C) receiving a frame of the bookshelf captured by the camera; D) performing image cutting on the frame to obtain the a back image of the book on the shelf; and E) establishing an inventory data group corresponding to the book, wherein the inventory data group includes the identification information, the inventory status, and the back image of the book.
本發明之再一目的,即在提供一種圖書庫存管理方法。Still another object of the present invention is to provide a book inventory management method.
於是,本發明圖書庫存管理方法,適於藉由與一攝影裝置連接的一系統執行,該方法包含下列步驟。A)提供已建立的至少一庫存資料組,其中,該庫存資料組包括一庫存圖書之一書背影像;B)接收該攝影裝置所擷取之一書架的一圖框;C)對該書架的圖框進行影像切割,以取得該書架上之一圖書的一書背影像;D)利用該書架上之圖書的書背影像,與該庫存圖書的書背影像以進行比對;及E)根據步驟D)之比對結果產生該書架上之圖書的一庫存狀態。Thus, the book inventory management method of the present invention is adapted to be executed by a system connected to a photographing apparatus, the method comprising the following steps. A) providing at least one inventory data set that has been established, wherein the inventory data set includes a back image of an inventory book; B) receiving a frame of a book shelf captured by the camera device; C) the book shelf The frame is image-cut to obtain a back image of a book on the shelf; D) using the back image of the book on the shelf to compare with the back image of the book; and E) The result of the comparison of step D) produces an inventory status of the books on the shelf.
有關本發明之前述及其他技術內容、特點與功效,在以下配合參考圖式之一個較佳實施例的詳細說明中,將可清楚的呈現。The above and other technical contents, features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments.
參閱圖1,本發明圖書庫存管理系統1之較佳實施例,係透過網路與多台佈建在書架附近的攝影裝置2連接,其中,攝影裝置2為PTZ(pan、tilt、zoom)網路攝影機;而攝影裝置2的數量,與圖書館或書店內的環境參數(例如,室內空間大小、書架數量)相關。Referring to FIG. 1, a preferred embodiment of the book inventory management system 1 of the present invention is connected to a plurality of camera devices 2 disposed near the bookshelf via a network, wherein the camera device 2 is a PTZ (pan, tilt, zoom) network. The number of the photographing devices 2 is related to environmental parameters in the library or bookstore (for example, the size of the indoor space, the number of bookshelves).
本發明圖書庫存管理系統1包含一人機介面11、一參數處理單元12、電連接於該人機介面11與該參數處理單元12之一記憶單元13、電連接於該記憶單元13之一影像處理單元14,及電連接於該記憶單元13與該影像處理單元14之一庫存資料管理單元15。該人機介面11用以供使用者輸入及設定系統參數。該參數處理單元12用以處理與該圖書庫存管理系統1及攝影裝置2相關之參數。該影像處理單元14用以接收攝影裝置2所擷取之影像,並對其進行與後續庫存管理有關的影像處理;該影像處理單元14包括一幾何校正模組141、一切割模組142、一特徵擷取模組143,及一比對模組144。該庫存資料管理單元15用以根據該影像處理單元14之處理結果進行對應之圖書庫存管理,並將其結果儲存於該記憶單元13。The book inventory management system 1 of the present invention comprises a human machine interface 11, a parameter processing unit 12, a memory unit 13 electrically connected to the human machine interface 11 and the parameter processing unit 12, and an image processing electrically connected to the memory unit 13. The unit 14 is electrically connected to the memory unit 13 and the inventory data management unit 15 of the image processing unit 14. The human machine interface 11 is used for the user to input and set system parameters. The parameter processing unit 12 is configured to process parameters related to the book inventory management system 1 and the photographing device 2. The image processing unit 14 is configured to receive an image captured by the photographing device 2 and perform image processing related to subsequent inventory management. The image processing unit 14 includes a geometric correction module 141, a cutting module 142, and a The feature capture module 143 and a comparison module 144. The inventory data management unit 15 is configured to perform corresponding book inventory management according to the processing result of the image processing unit 14, and store the result in the memory unit 13.
在本較佳實施例中,該人機介面11包括供使用者操作及輸入資料之一週邊輸入裝置(例如,鍵盤、滑鼠,圖未示),及用於讀取圖書條碼的一掃描器(圖未示)。該參數處理單元12、該影像處理單元14,及該庫存資料管理單元15係整合成一軟體並安裝於一電腦以執行本發明圖書庫存管理方法。該記憶單元13係為一資料庫或其他儲存裝置。In the preferred embodiment, the human machine interface 11 includes a peripheral input device (for example, a keyboard, a mouse, not shown) for the user to operate and input data, and a scanner for reading the barcode of the book. (not shown). The parameter processing unit 12, the image processing unit 14, and the inventory data management unit 15 are integrated into a software and installed in a computer to execute the book inventory management method of the present invention. The memory unit 13 is a database or other storage device.
本發明圖書庫存管理方法可分為以下三個階段進行說明。The book inventory management method of the present invention can be divided into the following three stages for explanation.
參閱圖1、圖2與圖3,在該圖書庫存管理系統1開始進行上架階段及盤點階段之相關步驟前,必須先設定所需的參數(組),因為關於每一台攝影裝置2之設定與操作大致相同,故以下以單台攝影裝置2為例進行說明。Referring to FIG. 1, FIG. 2 and FIG. 3, before the book inventory management system 1 starts the relevant steps of the shelf stage and the inventory stage, the required parameters (groups) must be set because of the setting of each camera 2 Since the operation is substantially the same as that of the operation, the single imaging device 2 will be described below as an example.
首先,由於攝影裝置2係預先架設於已知位置,所以,攝影裝置2之一攝影裝置識別資訊(camera identification,以下簡稱CID)為預設之資訊;而攝影裝置2的拍攝範圍亦在其架設之後就已固定,所以,與其拍攝範圍內書架之圖框(frame)數目相關的一列(row)數及一行(column)數也可預知,將每一圖框3所在的列、行定義為一書架位置資訊;舉例來說,圖3為攝影裝置2的拍攝範圍內書架之所有圖框3,其對應的列數為3,行數為4,每一圖框3的書架位置資訊可表示為(列i ,行j )。而該攝影裝置識別資訊及該書架位置資訊係用於後續產生其中一圖書的一圖書位置資訊之用,在步驟S11中,使用者藉由該人機介面11之週邊輸入裝置,設定該攝影裝置識別資訊及該等書架位置資訊,且將其等儲存於該記憶單元13內。First, since the photographing device 2 is preliminarily mounted at a known position, one of the photographing devices 2 has a camera identification information (CID) as a preset information; and the photographing range of the photographing device 2 is also set up in the photographing device 2 It is fixed afterwards, so the number of rows and the number of columns related to the number of frames of the bookshelf in the shooting range can be predicted. The column and row in which each frame 3 is located is defined as one. For example, FIG. 3 is all the frame 3 of the bookshelf in the shooting range of the photographing device 2, and the corresponding column number is 3, the number of rows is 4, and the book position information of each frame 3 can be expressed as (column i , line j ). The photographic device identification information and the book position information are used for subsequently generating a book location information of one of the books. In step S11, the user sets the photographic device by using the peripheral input device of the human interface 11 The information and the bookshelf location information are identified and stored in the memory unit 13.
繼而,在步驟S12中,使用者移動攝影裝置2鏡頭至其拍攝範圍內四個角落的書架位置(例如,對應於圖3中的(1,1)、(1,4)、(3,1),及(3,4)四個圖框3的書架位置),該參數處理單元12取得對應於每一角落的書架位置之一PTZ參數組及多個控制點座標並將其等儲存於該記憶單元13內。在本較佳實施例中,每一圖框3對應真實世界座標系的四個控制點座標,及成像座標系統的四個控制點座標。有關PTZ參數組及控制點座標之細節係為熟習此項技術者所熟知,故不在此贅述。Then, in step S12, the user moves the lens of the photographing device 2 to the bookshelf position at four corners of the shooting range thereof (for example, corresponding to (1, 1), (1, 4), (3, 1 in FIG. 3). And (3, 4) the four bookshelf positions of the frame 3), the parameter processing unit 12 obtains one of the bookshelf positions corresponding to each corner and a plurality of control point coordinates and stores them in the Inside the memory unit 13. In the preferred embodiment, each frame 3 corresponds to four control point coordinates of the real world coordinate system and four control point coordinates of the imaging coordinate system. The details of the PTZ parameter set and control point coordinates are well known to those skilled in the art and will not be described here.
最後,在步驟S13中,該參數處理單元12利用內插法,計算攝影裝置2之拍攝範圍內其他書架位置中每一者之一PTZ參數組及多個控制點座標,並將其等儲存於該記憶單元13內。Finally, in step S13, the parameter processing unit 12 calculates one of the PTZ parameter sets and the plurality of control point coordinates of each of the other bookshelf positions in the shooting range of the photographing device 2 by using the interpolation method, and stores them in the Inside the memory unit 13.
值得一提的是,系統初始化階段的步驟並不需要經常執行,只需在書架或攝影裝置2的位置被移動過才需重新執行設定。It is worth mentioning that the steps of the system initialization phase do not need to be performed frequently, and only need to be re-executed when the position of the bookshelf or the photographing device 2 is moved.
參閱圖1、圖4與圖5A~圖5C,在系統初始化完成之後,使用者可透過該人機介面11之週邊輸入裝置將系統切換至一上架模式以開始進行上架階段之相關處理,其中,上架階段包括下列步驟。Referring to FIG. 1 , FIG. 4 , and FIG. 5A to FIG. 5C , after the system is initialized, the user can switch the system to a shelf mode through the peripheral input device of the human interface 11 to start the related processing in the shelf stage. The shelf stage includes the following steps.
如步驟S201所示,使用者藉由該人機介面11之掃描器,讀取每一欲上架的圖書之圖書條碼,每一圖書條碼對應一圖書的識別資訊,一般而言,圖書的識別資訊為國際標準圖書編號(International Standard Book Number,ISBN)。該圖書庫存管理系統1接收每一欲上架的圖書之識別資訊,並將其儲存於該記憶單元13。As shown in step S201, the user reads the bar code of each book to be put up by the scanner of the man-machine interface 11, and each bar code corresponds to the identification information of a book. Generally, the identification information of the book It is the International Standard Book Number (ISBN). The book inventory management system 1 receives the identification information of each book to be put on the shelf and stores it in the memory unit 13.
然後,使用者將欲上架的圖書依一預設的上架順序進行圖書上架。Then, the user puts the books to be put on shelves according to a preset shelf order.
如步驟S202所示,該庫存管理單元15初始化每一已上架的圖書之一庫存狀態,在本較佳實施例中,初始化的庫存狀態為「架上狀態」。As shown in step S202, the inventory management unit 15 initializes an inventory status of each of the books that have been placed. In the preferred embodiment, the initialized inventory status is "on-state".
如步驟S203所示,該庫存管理單元15以攝影裝置識別資訊、書架位置識別資訊,及上架順序,形成每一已上架的圖書之圖書位置資訊。As shown in step S203, the inventory management unit 15 forms the book location information of each book that has been put on the shelf by the camera identification information, the shelf location identification information, and the shelf order.
如步驟S204所示,該參數處理單元12提供預先儲存於該記憶單元13內的PTZ參數組,以透過網路控制攝影裝置2來擷取書架之圖框3。As shown in step S204, the parameter processing unit 12 provides a PTZ parameter set stored in advance in the memory unit 13 to control the photographing device 2 through the network to capture the frame 3 of the bookshelf.
由於該圖書庫存管理系統1對於每台攝影裝置2所擷取到的書架之任一圖框3的處理方式相同,所以,為了便於描述,在後續的說明書中係以書架的其中一圖框3進行說明。Since the book inventory management system 1 processes the same frame 3 of the bookshelf captured by each of the photographing devices 2, for the sake of convenience of description, in the subsequent description, one of the frames of the bookshelf is used. Be explained.
如步驟S205所示,因為攝影裝置2拍攝位置與角度的關係,使得擷取的圖框3之內容呈現歪斜的現象(如,圖5A),故必須對擷取的圖框3進行影像幾何校正。該影像處理單元14之幾何校正模組141利用預先建立的一圖框幾何轉換矩陣對圖框3進行影像幾何校正,其中,該圖框幾何轉換矩陣係由儲存於該記憶單元13內之圖框3對應的控制點座標預先建立,其中,關於該圖框幾何轉換矩陣之求法係為習知技術,可參考「”Digital Image Processing Using MATLAB”,R. C. Gonzalez and R. E. Woods,3rd Edition,Prentice Hall.」,故不在此贅述。As shown in step S205, because the photographing device 2 captures the relationship between the position and the angle, the content of the captured frame 3 is skewed (eg, FIG. 5A), so the image geometric correction must be performed on the captured frame 3. . The geometric correction module 141 of the image processing unit 14 performs image geometric correction on the frame 3 by using a pre-established frame geometry transformation matrix, wherein the frame geometry transformation matrix is framed in the memory unit 13 3 corresponding to the control point coordinates of a pre-established, wherein, with respect to the frame geometric transformation method for finding based matrices according to the prior art technique, reference may be "" Digital Image Processing Using MATLAB ", RC Gonzalez and RE Woods, 3 rd Edition, Prentice Hall. Therefore, I will not repeat them here.
如步驟S206所示,該影像處理單元14之切割模組142對已校正的圖框3' (如,圖5B)進行影像切割,以得到書架上之圖書的書背影像;其中,影像切割包括下列子步驟。As shown in step S206, the cutting module 142 of the image processing unit 14 performs image cutting on the corrected frame 3 ' (eg, FIG. 5B) to obtain a book back image of the book on the shelf; wherein the image cutting includes The following substeps.
a)對已校正的圖框3' 進行垂直邊緣偵測,並將邊緣化後的結果進行水平方向的直方圖(histogram)投影(projection)。a) Perform vertical edge detection on the corrected frame 3 ' and perform a histogram projection in the horizontal direction.
b)利用a)之執行結果進行判斷,若直方圖投影的結果大於預設的一門檻值(threshold),則表示其為圖書的標題(title),依此方式找出標題範圍的一上邊界及一下邊界。b) judging by using the execution result of a), if the result of the histogram projection is greater than a preset threshold, it indicates that it is the title of the book, and in this way, find an upper boundary of the title range. And the next boundary.
c)對已校正的圖框3' 進行水平邊緣偵測,並將邊緣化後的結果進行水平方向的直方圖投影。c) Perform horizontal edge detection on the corrected frame 3 ' and project the edged results in a horizontal histogram.
d)利用c)之執行結果,並配合已找出的標題範圍的上邊界及下邊界,找出可能的書架邊緣31,然後進行關於書架邊緣31的影像切割,以得到如圖5C所示的圖框3" 。d) using the execution result of c), and matching the upper and lower boundaries of the identified title range, finding the possible bookshelf edge 31, and then performing image cutting on the bookcase edge 31 to obtain as shown in FIG. 5C. Box 3 " .
其中,上述a)與c)中的垂直、水平邊緣偵測,係採用Sobel邊緣偵測法。Among them, the vertical and horizontal edge detection in a) and c) above are Sobel edge detection.
在繼續以下子步驟描述之前,先進行下列定義,以便於了解。定義一狀態轉換模型,該狀態轉換模型為一有限狀態機(finite state automation),具有六種相異的狀態類別,分別是左圖書邊緣(left boundary,以q 1 表示)、右圖書邊緣(right boundary,以q 2 表示)、左書脊(left spine,以q 3 表示)、右書脊(right spine,以q 4 表示)、標題(以q 5 表示),及背景(background,以q 6 表示),整理成一類別集合Q ={q 1 ,...,q 6 },圖框3" 的一中央水平線32上的多個位置點i 的狀態s i 分別對應到上述狀態類別其中一者,,而圖書擺放結構的最可能頂端位置以p i 表示。Before proceeding with the description of the sub-steps below, make the following definitions for easy understanding. A state transition model is defined. The state transition model is a finite state automation having six distinct state categories, namely a left boundary (represented by q 1 ) and a right book edge (right Boundary, denoted by q 2 ), left spine (represented by q 3 ), right spine (represented by q 4 ), title (indicated by q 5 ), and background (background, denoted by q 6 ) , organized into a set of classes Q = {q 1, ..., q 6}, the state s frame 3 'positions of a plurality of points 32 on a central horizontal line i-i respectively correspond to the above-described state wherein one category, And the most likely top position of the book placement structure is denoted by p i .
e)利用該狀態轉換模型並配合利用動態規劃(dynamic programming)演算法,以取得書架上之圖書的書背影像。e) Using the state transition model in conjunction with the use of a dynamic programming algorithm to obtain a book back image of a book on a shelf.
其中,有關該狀態轉換模型配合動態規劃演算法的描述,可參考E. Taira等人提出的「A model-based book boundary detection technique for bookshelf image analysis.」,及R. Bellman等人提出的「Applied Dynamic Programming.」。For a description of the state transition model and the dynamic programming algorithm, refer to "A model-based book boundary detection technique for bookshelf image analysis." by E. Taira et al., and "Applied" by R. Bellman et al. Dynamic Programming.".
值得一提的是,本發明更加入了對應位置點i 的多個色相(hue)函數值作為動態規劃演算法之權重,每一色相函數值與其對應位置點的色相平均值及狀態兩者中至少一者相關,色相函數d T [(s i ,p i )|(s i -1 ,p i -1 )]的定義如式1所示。It is worth mentioning that the present invention further adds a plurality of hue function values corresponding to the position point i as the weight of the dynamic programming algorithm, and the hue average value and state of each hue function value and its corresponding position point. At least one of them is related, and the definition of the hue function d T [( s i , p i )|( s i -1 , p i -1 )] is as shown in Equation 1.
其中,d H (H i ,H i -1 )的定義如式2所示。Wherein, d H ( H i , H i -1 ) is as defined in Formula 2.
d H (H i ,H i -1 )=1-cos(2×π×(H i -H i -1 )).................................式2而H i 為位置點i 至其頂端位置p i 連線上像素(pixel)的色相平均值。 d H ( H i , H i -1 )=1-cos(2×π×( H i - H i -1 )).................... . . . Equation 2 and H i is the hue average of the pixels on the line from the position point i to the top position p i .
如步驟S207所示,該影像處理單元14之幾何校正模組141還將步驟S206中e)切割出來的書背影像,利用一書背幾何轉換矩陣進行影像幾何校正,其中,影像幾何校正的方式類似於步驟S205;差別僅在於:其係先根據e)切割出來的書背影像的四個邊界,求出四個書背控制點座標;再根據真實世界座標系的四個控制點座標,及上述書背控制點座標求出該書背幾何轉換矩陣。As shown in step S207, the geometric correction module 141 of the image processing unit 14 further cuts the back image of the book cut in step S206 by using a back geometry conversion matrix, where the image geometric correction is performed. Similar to step S205; the difference is only that: according to the four boundaries of the back image of the book cut out e), the coordinates of the four book back control points are obtained; and then according to the coordinates of the four control points of the real world coordinate system, and The book back control point coordinates are used to find the book back geometry transformation matrix.
由步驟S206切割出來的書背影像,仍然有可能因為下列三種情況而造成誤判,所以本發明更對書背影像進行後處理,以得到更為精確的切割結果。The back image of the book cut out in step S206 may still be misjudged due to the following three situations, so the present invention further post-processes the back image to obtain a more accurate cutting result.
情況一、非書背影像(如,圖書間的間隙、書櫃隔層)誤判成書背影像;情況二、書背影像的標題太少而誤判為背景;及情況三、多本圖書誤判為一本,此情況常出現在同系列或多本書背顏色相同的圖書擺放在一起。Case 1, non-bookback images (eg, gaps between books, bookcase compartments) are misjudged into the back of the book; situation 2, the title of the book back is too small and misjudged as the background; and the situation 3, many books misjudged as one Ben, this situation often appears in the same series or multiple books with the same color on the back of the book.
如步驟S208所示,對應上述三種情況,該影像處理單元14之切割模組142對已校正的書背影像進行後處理,以進一步地切割已校正的書背影像。其中,後處理包括下列子步驟。As shown in step S208, in accordance with the above three cases, the cutting module 142 of the image processing unit 14 performs post-processing on the corrected book back image to further cut the corrected book back image. Among them, the post-processing includes the following sub-steps.
f)對書背影像進行水平邊緣偵測,並將邊緣化後的結果進行垂直方向的直方圖投影。f) Perform horizontal edge detection on the back image of the book, and project the edged result into a histogram projection in the vertical direction.
g)利用f)的執行結果,進行對於上述三種情況的判斷:若執行結果對應之書背影像的寬度小於預設的一書背寬門檻值,則將其視為非書背影像;若執行結果未對應出標題,將其視為非書背影像;若執行結果對應出書背影像內的一標題數目大於一,則可進一步求出書背影像內每一標題之寬度。g) using the execution result of f), performing the judgment on the above three cases: if the width of the book back image corresponding to the execution result is smaller than the preset book back width threshold, it is regarded as a non-book back image; The result is not corresponding to the title, and is regarded as a non-bookback image; if the execution result corresponds to the number of titles in the back image of the book being greater than one, the width of each title in the back image can be further determined.
h)根據每一標題之寬度進一步地進行影像切割。h) further image cutting according to the width of each title.
如步驟S209所示,該影像處理單元14的特徵擷取模組143對步驟S208進一步切割後的書背影像進行尺度不變轉換(Scale Invariant Feature Transform,SIFT),以得到圖書的至少一書背特徵。其中,尺度不變轉換係為習知技術,其細節可參考D. G. Lowe提出的「Object recognition from local scale-invariant features.」,故不在此贅述。As shown in step S209, the feature capturing module 143 of the image processing unit 14 performs Scale Invariant Feature Transform (SIFT) on the book back image further cut in step S208 to obtain at least one book back of the book. feature. Among them, the scale-invariant conversion system is a conventional technique, and the details thereof can be referred to "Object recognition from local scale-invariant features." proposed by D. G. Lowe, and therefore will not be described here.
如步驟S210所示,該庫存資料管理單元15建立對應已上架的每一圖書之一庫存資料組,並將庫存資料組儲存至該記憶單元13;其中;每一庫存資料組包括其識別資訊、庫存狀態、圖書位置資訊、書背影像,及書背特徵。As shown in step S210, the inventory data management unit 15 establishes an inventory data group corresponding to each book that has been put on the shelf, and stores the inventory data group to the memory unit 13; wherein each inventory data group includes its identification information, Inventory status, book location information, book back image, and book back features.
參閱圖4與圖6,值得一提的是,在上架階段中,該圖書庫存管理系統1還可以先根據每一圖書的識別資料(如,上述的ISBN),由網路自動擷取每一圖書的一詮釋資料(metadata),每一詮釋資料包括該圖書的實際寬度與實際高度,還有該圖書的精裝本或平裝本與頁數(可用於推算圖書的實際厚度)。由於上架階段可以明確地知道對應每一圖書的該書架位置識別資訊、上架順序,並已知圖書位置資訊;如圖6所示,若所有圖書皆直立且緊密地排列於書架上,則可以下述之完美切割程序,自動地進行完美切割,取得最準確的書背影像,並累積此等書背影像之相關資訊,作為後續盤點階段之用。該完美切割程序包括下列子步驟。Referring to FIG. 4 and FIG. 6, it is worth mentioning that, in the shelf stage, the book inventory management system 1 can also automatically retrieve each of the books according to the identification data of each book (for example, the above-mentioned ISBN). A meta-data of a book, each interpretation material including the actual width and actual height of the book, as well as the hardcover or paperback of the book and the number of pages (which can be used to estimate the actual thickness of the book). Since the shelf stage can clearly know the book position identification information and the shelf order corresponding to each book, and the book location information is known; as shown in FIG. 6, if all the books are erect and closely arranged on the bookshelf, then the next The perfect cutting program, which automatically cuts perfectly, obtains the most accurate back image of the book, and accumulates information about the back image of these books, which is used as a subsequent inventory stage. This perfect cutting procedure includes the following sub-steps.
i)將每一圖書的實際高度與實際厚度轉換到與圖框3" (如圖6所示)相同之影像座標系統,以得到每一圖書之長度與寬度(以像素為單位)。i) converting the actual height of each book and the actual thickness of the same image coordinate system "(shown in FIG frame 36), to afford the length and width of each book (in pixels).
j)利用類似於上述步驟206之d)的執行結果,進行水平邊緣偵測,並將邊緣化後的結果進行垂直方向的直方圖投影。j) Using the execution result similar to d) of the above step 206, performing horizontal edge detection and projecting the edged result into a histogram projection in the vertical direction.
k)利用j)之執行結果進行判斷,若直方圖投影的結果大於預設的一門檻值,則表示其為圖書的標題之正確位置,依此方式找出該圖框3" 中每一圖書的標題範圍。k) using j) of the results of the determination, if the result of the histogram of a projection is greater than a predetermined threshold value, it indicates that, in this manner to find the correct position of the frame 3, the title of the book "each Books The scope of the title.
l)利用k)之執行結果,找出每一圖書的標題範圍之一正中央位置,由此正中央位置往左、右兩方向延伸相同之一距離Dk ,此距離Dk 為i)所得寬度的一半,依此方式找出該圖書的一左邊界及一右邊界並根據此一左邊界及右邊界進行影像切割。l) Using the execution result of k), find out the positive central position of one of the title ranges of each book, whereby the positive central position extends to the left and right directions by the same distance D k , and the distance D k is i) Half of the width, in this way, find a left boundary and a right boundary of the book and perform image cutting according to the left and right borders.
以上子步驟i)~l),主要原則在於「利用所有圖書的詮釋資料,累積每一圖書之實際長度、寬度、厚度與影像座標系統之像素的比例範圍之相關資訊」。當上述子步驟e)到h)的準確率到可用的範圍(90%以上),該完美切割程序有利於將來在盤點時,提昇切割的準確率,以補足上述子步驟e)到h)未臻完美之處,使得某些未被準確辨識切割的書背影像(因為,盤點時圖書擺放的位置與狀態較為複雜),能夠進一步地利用上述完美切割程序被切割出來。The above sub-steps i)~l), the main principle is to "use the interpretation data of all books to accumulate information about the actual length, width, thickness of each book and the proportion range of pixels of the image coordinate system". When the accuracy of the above sub-steps e) to h) is within the usable range (more than 90%), the perfect cutting procedure facilitates the improvement of the cutting accuracy in the future when the inventory is taken to complement the above sub-steps e) to h) The perfect point is that some of the book back images that are not accurately identified and cut (because the position and state of the book placed at the time of inventory are more complicated) can be further cut out using the above perfect cutting program.
上述中更值得一提的是我們也可以設計一圖書掃瞄機器(嵌入式系統含硬體),在上架時量測圖書之實際長度、寬度、厚度,並且結合條碼掃瞄機及攝影裝置,以一併取得圖書之條碼與書背影像,使得圖書的詮釋資料更符合實際情形並且達到完整。It is worth mentioning that we can also design a book scanning machine (embedded system with hardware), measure the actual length, width and thickness of the book when it is put on the shelf, and combine the barcode scanner and the photographic device. By obtaining the barcode of the book together with the image of the back of the book, the interpretation data of the book is more in line with the actual situation and is complete.
參閱圖1、圖5A~圖5C,與圖6,當圖書被上架之後即為庫存圖書,該記憶單元13儲存有已建立的多筆庫存資料組,使用者可透過該人機介面11之週邊輸入裝置將系統切換至一盤點模式以開始進行盤點階段之相關處理,其中,盤點階段包括下列步驟。Referring to FIG. 1 , FIG. 5A to FIG. 5C , and FIG. 6 , when the book is put on the shelf, it is an inventory book, and the memory unit 13 stores the established multiple inventory data groups, and the user can access the periphery of the human machine interface 11 . The input device switches the system to an inventory mode to begin the relevant processing of the inventory phase, wherein the inventory phase includes the following steps.
如步驟S31所示,該參數處理單元12提供預先儲存於該記憶單元13內的PTZ參數組,以透過網路控制攝影裝置2來擷取書架之圖框3。As shown in step S31, the parameter processing unit 12 provides a PTZ parameter set stored in advance in the memory unit 13 to control the photographing device 2 through the network to capture the frame 3 of the bookshelf.
如步驟S32所示,該影像處理單元14之幾何校正模組141對圖框3進行影像幾何校正,其校正方式類似於步驟S205,故不在此贅述。As shown in step S32, the geometric correction module 141 of the image processing unit 14 performs image geometric correction on the frame 3, and the correction manner is similar to the step S205, and thus will not be described herein.
如步驟S33所示,該影像處理單元14之切割模組142對已校正的圖框3' 進行影像切割,以得到書架上之圖書的書背影像,其影像切割的方式類似於步驟S206,故不在此贅述。As shown in step S33, the cutting module 142 of the image processing unit 14 performs image cutting on the corrected frame 3 ' to obtain a book back image of the book on the shelf, and the image cutting manner is similar to step S206. Not to repeat here.
如步驟S34所示,該影像處理單元14之幾何校正模組141對步驟S33切割出來的書背影像進行影像幾何校正,其校正方式類似於步驟S207,故不在此贅述。As shown in step S34, the geometric correction module 141 of the image processing unit 14 performs image geometric correction on the book back image cut in step S33, and the correction manner is similar to step S207, and thus will not be described herein.
如步驟S35所示,該影像處理單元14之切割模組142對步驟S34校正後的書背影像進行後處理,以進一步地切割已校正的書背影像,其後處理方式類似於步驟S208,故不在此贅述。As shown in step S35, the cutting module 142 of the image processing unit 14 performs post-processing on the back image corrected in step S34 to further cut the corrected book back image, and the processing manner is similar to step S208. Not to repeat here.
如步驟S36所示,該影像處理單元14之特徵擷取模組143對步驟S35後處理後的書背影像進行尺度不變特徵轉換,以得到書架上圖書的書背特徵,其轉換方式類似於步驟S209,故不在此贅述。As shown in step S36, the feature capturing module 143 of the image processing unit 14 performs scale-invariant feature conversion on the back image processed after step S35 to obtain the book back feature of the book on the shelf, and the conversion manner is similar to Step S209, therefore, it will not be described here.
如步驟S37所示,該影像處理單元14之比對模組144將該書架上之圖書的書背影像,與庫存圖書的書背影像進行比對;在本較佳實施例中,該比對模組144係將書架上圖書的書背特徵,與庫存圖書的書背特徵進行特徵比對。As shown in step S37, the comparison module 144 of the image processing unit 14 compares the back image of the book on the bookshelf with the back image of the book; in the preferred embodiment, the comparison The module 144 compares the book back features of the books on the bookshelf with the book back features of the stock books.
如步驟S38所示,該庫存管理單元15根據比對結果,產生書架上圖書的庫存狀態。在本較佳實施例中,庫存狀態可為「架上狀態」、「新增狀態」,及「遺失或售出狀態」三種狀態其中一者。定義在盤點階段中書架上的圖書之書背影像為查詢影像集A ={a 1 ,a 2 ,...,a n },該記憶單元13內的庫存圖書之書背影像為庫存影像集D ={d 1 ,d 2 ,...,d m },上述三種狀態之定義可整理為表一。As shown in step S38, the inventory management unit 15 generates an inventory status of the books on the bookshelf based on the comparison result. In the preferred embodiment, the inventory status may be one of three states: "on-state", "new state", and "lost or sold state". The back image of the book defined on the bookshelf in the inventory stage is the query image set A = { a 1 , a 2 , ..., a n }, and the back image of the book in the memory unit 13 is the stock image set. D = { d 1 , d 2 , ..., d m }, the definitions of the above three states can be organized into Table 1.
對於架上狀態的圖書,該庫存管理單元15還可以根據影像集A 和D 的序列,分析哪些圖書位置亂了;並將分析結果提供給使用者,供其以人工方式將圖書排回原位置,以盡量維持圖書原始上架時的各種狀態,然後系統可重新回到盤點階段的第一步驟S31。For the book on the shelf status, the inventory management unit 15 can also analyze which book positions are disordered according to the sequence of the image sets A and D ; and provide the analysis result to the user for manually returning the book to the original position. In order to maintain the various states when the book is originally put on the shelf, the system can then return to the first step S31 of the inventory phase.
如步驟S39所示,該庫存管理單元15建立對應書架上之每一圖書之一盤點資料組,並將盤點資料組儲存至該記憶單元13;其中;每一盤點資料組包括其庫存狀態、書背影像,及書背特徵。值得一提的是,盤點資料組還可進一步提供給使用者作為參考,以使其了解目前圖書的各種狀態。As shown in step S39, the inventory management unit 15 creates an inventory data group corresponding to each book on the bookshelf, and stores the inventory data group to the memory unit 13; wherein each inventory data group includes its inventory status, book Back image, and book back features. It is worth mentioning that the inventory data group can be further provided to the user as a reference to make them aware of the various states of the current book.
在實際應用上,本發明亦可在使用者欲找尋某本圖書時,即時地提供該圖書的目前影像資料,舉例來說,使用這輸入該本圖書之識別資訊至該圖書庫存管理系統1,該庫存管理單元15根據所輸入之識別資訊由該記憶單元13得知該圖書之圖書位置資訊,及其對應的PTZ參數組,即可接著由該參數處理單元12提供該PTZ參數組,以控制攝影裝置即時地取得該圖書的目前影像資料,類似於此等應用,皆可藉由本發明所提出的該圖書庫存管理系統1及其方法來達成,並不限於本較佳實施例所揭露。In practical applications, the present invention can also provide the current image data of the book in real time when the user wants to find a certain book, for example, using the input information of the book to input the book inventory management system 1, The inventory management unit 15 is informed by the memory unit 13 of the book location information of the book and its corresponding PTZ parameter group according to the input identification information, and then the PTZ parameter group is provided by the parameter processing unit 12 to control The photographing device can obtain the current image data of the book in real time, and the application is similar to the application, and can be achieved by the book inventory management system 1 and the method thereof provided by the present invention, and is not limited to the preferred embodiment.
綜上所述,本發明所提出的該圖書庫存管理系統1及其方法,係利用現有書架附近的攝影裝置2,進行以視覺為基礎之圖書的自動化庫存管理,不但降低了人事成本,亦避免了習知以RFID技術進行圖書管理所遭遇的缺點;再者,本發明中所使用的書背影像切割技術,更配合了色相平均值作為參考,且加上後處理,大幅提昇書背影像切割的精確度,此亦有助於提高系統的執行效能與管理能力;故確實能達成本發明之目的。In summary, the book inventory management system 1 and method thereof according to the present invention utilizes the photographic device 2 near the existing bookshelf to perform automated inventory management of the visual-based book, which not only reduces personnel costs, but also avoids The shortcomings encountered in the management of books by RFID technology; in addition, the book back image cutting technology used in the present invention is further matched with the hue average as a reference, and post-processing is added to greatly improve the book back image cutting. The accuracy of this also helps to improve the performance and management capabilities of the system; therefore, the object of the present invention can be achieved.
惟以上所述者,僅為本發明之較佳實施例而已,當不能以此限定本發明實施之範圍,即大凡依本發明申請專利範圍及發明說明內容所作之簡單的等效變化與修飾,皆仍屬本發明專利涵蓋之範圍內。The above is only the preferred embodiment of the present invention, and the scope of the invention is not limited thereto, that is, the simple equivalent changes and modifications made by the scope of the invention and the description of the invention are All remain within the scope of the invention patent.
1‧‧‧圖書庫存管理系統1‧‧‧Book Inventory Management System
11‧‧‧人機介面11‧‧‧Human Machine Interface
12‧‧‧參數處理單元12‧‧‧Parameter Processing Unit
13‧‧‧記憶單元13‧‧‧ memory unit
14‧‧‧影像處理單元14‧‧‧Image Processing Unit
141‧‧‧幾何校正模組141‧‧‧Geometry Correction Module
142‧‧‧切割模組142‧‧‧ cutting module
143‧‧‧特徵擷取模組143‧‧‧Feature capture module
144‧‧‧比對模組144‧‧‧ comparison module
15‧‧‧庫存資料管理單元15‧‧‧Stock Data Management Unit
2‧‧‧攝影裝置2‧‧‧Photographing device
S11~S13‧‧‧步驟S11~S13‧‧‧Steps
3‧‧‧圖框3‧‧‧ frame
S201~S210‧‧‧步驟S201~S210‧‧‧Steps
3' ‧‧‧已校正的圖框3 ' ‧‧‧ corrected frame
3" ‧‧‧已進行書架邊緣的影像切割後之圖框3 " ‧‧‧ Frames after image cutting on the edge of the bookshelf
31‧‧‧書架邊緣31‧‧‧ bookshelf edge
32‧‧‧中央水平線32‧‧‧Central horizontal line
Dk ‧‧‧距離D k ‧‧‧Distance
S31~S39‧‧‧步驟S31~S39‧‧‧Steps
圖1是一方塊圖,說明本發明圖書庫存管理系統之一較佳實施例;1 is a block diagram showing a preferred embodiment of the book inventory management system of the present invention;
圖2是一流程圖,說明本發明圖書庫存管理方法之一系統初始化階段;2 is a flow chart illustrating a system initialization phase of one of the book inventory management methods of the present invention;
圖3是一示意圖,說明一攝影裝置的拍攝範圍內書架之所有圖框;Figure 3 is a schematic view showing all the frames of the bookshelf in the shooting range of a photographing device;
圖4是一流程圖,說明本發明圖書庫存管理方法之一上架階段;4 is a flow chart showing one stage of the book inventory management method of the present invention;
圖5A~5C是一示意圖,說明其中一圖框之影像幾何校正及影像切割;5A-5C are schematic diagrams illustrating image geometric correction and image cutting of one of the frames;
圖6是一示意圖,說明圖書皆直立且緊密地排列於書架上之圖框;及Figure 6 is a schematic view showing the frames in which the books are erect and closely arranged on the bookshelf; and
圖7是一流程圖,說明本發明圖書庫存管理方法之一盤點階段。Figure 7 is a flow chart showing an inventory stage of the book inventory management method of the present invention.
1...圖書庫存管理系統1. . . Book inventory management system
11...人機介面11. . . Human machine interface
12...參數處理單元12. . . Parameter processing unit
13...記憶單元13. . . Memory unit
14...影像處理單元14. . . Image processing unit
141...幾何校正模組141. . . Geometric correction module
142...切割模組142. . . Cutting module
143...特徵擷取模組143. . . Feature capture module
144...比對模組144. . . Alignment module
15...庫存資料管理單元15. . . Inventory data management unit
2...攝影裝置2. . . Photography device
Claims (29)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW098145906A TWI423154B (en) | 2009-12-30 | 2009-12-30 | Book Inventory Management System and Its Method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW098145906A TWI423154B (en) | 2009-12-30 | 2009-12-30 | Book Inventory Management System and Its Method |
Publications (2)
Publication Number | Publication Date |
---|---|
TW201123041A TW201123041A (en) | 2011-07-01 |
TWI423154B true TWI423154B (en) | 2014-01-11 |
Family
ID=45046500
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW098145906A TWI423154B (en) | 2009-12-30 | 2009-12-30 | Book Inventory Management System and Its Method |
Country Status (1)
Country | Link |
---|---|
TW (1) | TWI423154B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110490169A (en) * | 2019-08-26 | 2019-11-22 | 阿里巴巴集团控股有限公司 | A kind of article rack method for inspecting, device and equipment |
CN113116229B (en) * | 2020-01-16 | 2022-11-29 | 深圳绿米联创科技有限公司 | Robot control method and device, sweeping robot and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TW367452B (en) * | 1994-03-14 | 1999-08-21 | Sade Technology Co | A system for implanting an image into a video stream |
US7124094B1 (en) * | 1999-10-27 | 2006-10-17 | Konica Corporation | Print system, service system, data server, master server, print client system and printer |
TW200639691A (en) * | 2005-05-04 | 2006-11-16 | Newsoft Technology Corp | System, method and medium of automatic document classification |
TW200939143A (en) * | 2008-03-14 | 2009-09-16 | Tlj Intertech Inc | A data management system |
-
2009
- 2009-12-30 TW TW098145906A patent/TWI423154B/en not_active IP Right Cessation
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TW367452B (en) * | 1994-03-14 | 1999-08-21 | Sade Technology Co | A system for implanting an image into a video stream |
US7124094B1 (en) * | 1999-10-27 | 2006-10-17 | Konica Corporation | Print system, service system, data server, master server, print client system and printer |
TW200639691A (en) * | 2005-05-04 | 2006-11-16 | Newsoft Technology Corp | System, method and medium of automatic document classification |
TW200939143A (en) * | 2008-03-14 | 2009-09-16 | Tlj Intertech Inc | A data management system |
Also Published As
Publication number | Publication date |
---|---|
TW201123041A (en) | 2011-07-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Xu et al. | Automatic extraction of 1D barcodes from video scans for drone-assisted inventory management in warehousing applications | |
Zimmermann et al. | Tracking by an optimal sequence of linear predictors | |
WO2019165892A1 (en) | Automatic vending method and apparatus, and computer-readable storage medium | |
US9805331B2 (en) | Smartphone-based asset management system | |
JP6549558B2 (en) | Sales registration device, program and sales registration method | |
US11030436B2 (en) | Object recognition | |
US11983250B2 (en) | Item-customer matching method and device based on vision and gravity sensing | |
Chen et al. | Building book inventories using smartphones | |
JP6036824B2 (en) | Angle of view variation detection device, angle of view variation detection method, and field angle variation detection program | |
US20150347801A1 (en) | Object recognition for exception handling in automatic machine-readable symbol reader systems | |
CN105528604A (en) | Bill automatic identification and processing system based on OCR | |
KR20120130788A (en) | Method and arrangement for multi-camera calibration | |
WO2020233221A1 (en) | Heatmap analysis method, device, and system | |
JP6684475B2 (en) | Image processing apparatus, image processing method and program | |
CN111080493A (en) | Dish information identification method and device and dish self-service settlement system | |
JP5674933B2 (en) | Method and apparatus for locating an object in a warehouse | |
CN111428743A (en) | Commodity identification method, commodity processing device and electronic equipment | |
KR20240058827A (en) | System and method for finding and classifying lines in an image with a vision system | |
TWI423154B (en) | Book Inventory Management System and Its Method | |
WO2012133371A1 (en) | Image capture position and image capture direction estimation device, image capture device, image capture position and image capture direction estimation method and program | |
JP2022033149A (en) | Information processing device, information processing method, and program | |
KR101982258B1 (en) | Method for detecting object and object detecting apparatus | |
US8622284B1 (en) | Determining and recording the locations of objects | |
WO2019181441A1 (en) | Information processing device, control method, program | |
CN113689458B (en) | 2D shooting track path calculation method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
MM4A | Annulment or lapse of patent due to non-payment of fees |