JP2021117920A - Storage device - Google Patents

Abstract

To provide a storage device that allows documents in a storage to be recognized easily.SOLUTION: A storage device includes: a storage that stores a document; an imaging unit disposed on an inner surface of the storage and captures an image of the cover of the document that is put in or taken out; a generation unit that generates a feature quantity of the put-in document based on the cover image captured by the imaging unit; and a registration unit that registers the feature quantity of the put-in document generated by the generation unit in association with an entry flag indicating the document is put in. Preferably, the storage device further includes a position detector for detecting an entry position of the document in the storage.SELECTED DRAWING: Figure 1

Description

The present technology relates to a storage device for storing documents such as books, files, drawings and envelopes.

Documents such as books, files, drawings and envelopes are stored in bookshelves in offices or library libraries. Documents are shared and moved to various places. Therefore, the location of the desired document may be unknown.

Patent Document 1 discloses an image processing device in which a camera is installed on a space, for example, a shelf and the ceiling of a room in which a document is placed, and the image processing device recognizes the putting in and out of the document on the shelf. The image processing device extracts text information from the image of the document captured by the camera, identifies the document, and recognizes the loading and unloading of the identified document on the shelf.

Japanese Unexamined Patent Publication No. 2019-28897

The image processing device recognizes a shelf containing desired documents by a camera installed in the room. However, when the camera is installed on the ceiling of a room or the like, the distance between the camera and the document is long, and it is difficult to identify the document from the captured image depending on the resolution of the camera.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide a storage device that makes it easy to recognize documents in the storage.

The storage device according to an embodiment of the present disclosure includes a storage for storing documents, an imaging unit arranged on the inner side surface of the storage and imaging the cover of the document to be stored or delivered, and the imaging unit. Based on the captured cover image, the generation unit that generates the feature amount of the document to be stored and the feature amount of the document generated and stored by the generation unit are registered in association with the receipt flag indicating the receipt. It has a registration unit to do.

In the storage device according to the embodiment of the present disclosure, an imaging unit is arranged inside the storage, and a feature amount of a document is generated based on a cover image captured by the imaging unit. Since the image pickup unit is provided in the storage, the distance between the document to be received or delivered and the image pickup section is shortened, and the document can be easily recognized from the image. Further, since the warehousing flag is associated with the feature amount of the generated document and registered, the user can recognize the document in the storage by referring to the registration information.

It is a schematic perspective view of the storage which concerns on Embodiment 1. FIG. It is a block diagram which shows the structure in the vicinity of a control device. It is a flowchart explaining the imaging process by a control device. It is a figure which shows an example of the cover image of the cover of a document. It is a figure which shows an example of the cover image of the back cover of a document. It is explanatory drawing explaining the distinction of warehousing or warehousing. It is a flowchart explaining the character image extraction process by a control device. It is explanatory drawing explaining the contour tracking. It is explanatory drawing explaining the rectangular area. It is a flowchart explaining the data calculation process by a control device. It is a conceptual diagram which shows an example of the extracted character image. It is a conceptual diagram which shows an example of size classification. It is a conceptual diagram which shows an example of the coordinate classification of a row with respect to the cover image of the front cover of FIG. It is a conceptual diagram which shows an example of the warehousing / delivery table stored in the storage part. It is a conceptual diagram which shows an example of the registration table stored in the storage part. It is a flowchart explaining the update process by a control device. It is a schematic diagram which shows an example of the display screen which displays the information of a document. It is a schematic diagram which shows the plurality of rectangular areas set for the character which has the plurality of figures which concerns on Embodiment 2. FIG. It is a schematic diagram which shows a plurality of circular areas set for the character which has a plurality of figures. It is a flowchart explaining the character image extraction process by a control device. It is a block diagram which shows the structure of the storage device which concerns on Embodiment 3. It is explanatory drawing which shows the structural example of the cover estimation part schematically. It is a flowchart explaining the update process by a server. It is a block diagram which shows the structure of the storage device which concerns on Embodiment 4. It is explanatory drawing which shows the structural example of the character estimation part schematically. It is a conceptual diagram which shows an example of the warehousing / delivery table. It is a conceptual diagram which shows an example of a character table. It is a conceptual diagram which shows an example of a registration table. It is a conceptual diagram which shows an example of a character table. It is explanatory drawing which shows the structural example of the cover estimation part schematically. It is a schematic diagram which shows an example of the display screen which displays the information of a document. It is a schematic diagram which shows an example of the display screen which displays the information of the document stored in a plurality of storages.

(Embodiment 1)
Hereinafter, the present invention will be described with reference to the drawings showing the storage device according to the first embodiment. FIG. 1 is a schematic perspective view of the storage. In the following description, the top, bottom, front, back, left, and right shown in the figure are used. The storage device includes a storage 1 for storing documents such as books or files. The storage 1 is, for example, a rectangular parallelepiped having an opening formed in the front surface. The storage 1 has a rectangular bottom plate 2, and two opposite side plates 3 are provided on the left and right sides of the bottom plate 2. The side plate 3 forms a rectangle extending vertically.

A rectangular back plate 4 facing the opening is arranged between the two side plates 3. The back plate 4 is connected to the bottom plate 2 and the rear sides of the two side plates 3. The top plate 5 facing the bottom plate 2 is connected to the back plate 4 and the upper sides of the two side plates 3. Inside the storage 1, the two side plates 3 support a plurality of shelf plates parallel to the top plate 5 and the bottom plate 2. A plurality of shelves are arranged in the vertical direction. The front side of the shelf plate is located on the rear side of the front side of the side plate 3, the top plate 5, and the bottom plate 2. One or more drawers 7 are provided on the lower side of the bottom plate 2.

A plurality of cameras 9 are provided on the inner surface of each of the two side plates 3. The plurality of cameras 9 are provided between the two shelves, between the shelves and the top plate 5, and between the shelves and the bottom plate 2, respectively. The camera 9 provided on the right side plate 3 is directed to the left side plate 3, and the camera 9 provided on the left side plate 3 is directed to the right side plate 3. The camera 9 is arranged between the front side of the side plate 3 and the front side of the shelf plate, and can take an image of the front cover and the back cover of the documents to be taken in and out.

Further, a plurality of position detectors 10 are provided on the right side plate 3. The position detector 10 is a non-contact proximity sensor such as an optical sensor or an ultrasonic sensor. The plurality of position detectors 10 are provided between the two shelves, between the shelves and the top plate 5, and between the shelves and the bottom plate 2, respectively. The position detector 10 is arranged between the front side of the side plate 3 and the front side of the shelf plate, and can detect the position of the document to be stored, in other words, which shelf plate the product is stored in. .. The position detector 10 may be provided on the left side plate 3 or on the two left and right side plates 3. The side plate 3 may be provided with illumination for imaging by the camera 9.

Two doors 8 with double doors are provided in the opening. The two doors 8 are rotatably attached to the two side plates 3 via hinges 11. Two open / close detectors 12 are provided on the front side of the top plate 5. The two open / close detectors 12 are arranged so as to face each of the two closed doors 8. The open / close detector 12 is, for example, a non-contact proximity sensor such as an optical sensor or an ultrasonic sensor, or a contact-type switch.

FIG. 2 is a block diagram showing a configuration near the control device 20. The storage device includes a control device 20. The control device 20 includes a control unit 21, a storage unit 22, a calculation unit 23, a timer 24, an image processing unit 25, and a thumbnail image generation unit 26. An image of the captured document is input from the camera 9 to the control device 20 via an interface (not shown), the position of the stored document is input from the position detector 10, and a signal indicating that the door 8 is opened (open signal). ) Or a signal indicating that it is closed (closed signal) is input from the open / close detector 12. The control unit 21 outputs a signal indicating the start or end of imaging to the camera 9 via the interface. The external device 30, for example, a personal computer, a smartphone or a tablet terminal can be connected to the control device 20 via an interface. The control unit 21 can output data to the external device 30 in response to a request from the external device 30. The interface is a wired or wireless interface. A touch panel, a display screen, and a keyboard may be attached to the storage 1 and used as the external device 30.

The control unit 21, the calculation unit 23, the image processing unit 25, and the thumbnail image generation unit 26 are, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or a logic circuit (for example, FPGA). It is composed of such as. The storage unit 22 is, for example, a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) or an EPROM (Erasable Programmable Read Only Memory), or a rewritable storage medium such as a hard disk.

The image processing unit 25 extracts a character image from the input document image. The calculation unit 23 calculates various information based on the extracted character image, the detection result of the position detector 10, and the like. The storage unit 22 stores a control program, a warehousing / delivery table described later, a registration table, and the like. The thumbnail image generation unit 26 generates a thumbnail image based on the input document image. The control unit 21 executes image processing and calculation described later based on the control program. At least a part of the control device 20 may be provided in a server accessible via a network.

The control device 20 executes an imaging process, a character image extraction process, a data calculation process, and an update process, which will be described later. First, the imaging process will be described. FIG. 3 is a flowchart illustrating an imaging process by the control device 20, FIG. 4 is a diagram showing an example of a cover image of the front cover of a document, and FIG. 5 is a diagram and a diagram showing an example of a cover image of the back cover of a document. 6 is an explanatory diagram for explaining the distinction between warehousing and warehousing.

As shown in FIG. 3, the control unit 21 determines whether or not an open signal has been input from the open / close detector 12 (S1). If no open signal is input (S1: NO), the process returns to S1. When the open signal is input (S1: YES), the control unit 21 outputs an imaging start signal to the camera 9 (S2). At this time, the camera 9 captures the cover images of the front and back covers of the documents to be received or delivered. The position detector 10 detects the position of the document to be received or delivered.

The control unit 21 acquires the front and back cover images (see FIGS. 4 and 5) from the camera 9, the position from the position detector 10, and the time captured from the timer 24 (S3). .. The front and back cover images at the same time are stored in the storage unit 22. The cover image is stored as a binary image.

The control unit 21 determines whether or not a closing signal has been input from the open / close detector 12 (S4). When the closing signal is not input (S4: NO), the control unit 21 returns the processing to S3. When the closing signal is input (S4: YES), the control unit 21 outputs an imaging end signal to the camera 9 (S5). The camera 9 finishes imaging. The camera 9 continues imaging while the door 8 is open, and continuously captures a front cover image for each of the front and back covers. The control unit 21 stores the warehousing / delivery identifier in the warehousing / delivery table (see FIG. 14) of the storage unit 22 (S6), links the warehousing / delivery identifier to the warehousing / delivery identifier, and acquires the front cover and the back cover in S3, respectively. The continuous cover image, time, and position of the above are stored (S7). The time is associated with the cover image. In FIG. 14, the description of the time is omitted.

The control unit 21 determines whether or not the document has been received based on the captured cover image (S8). The control unit 21 confirms, for example, a change in the position of a plurality of cover images of the front cover stored in the storage unit 22 over time. When the position of the cover image changes from the front side to the rear side as shown by the white arrow in FIG. 6, it is determined to be warehousing, and as shown by the solid arrow in FIG. 6, the position of the cover image changes from the rear side to the front side. If it changes to, it is judged that the goods have been issued. It should be noted that the change in the position of the front cover image of the back cover may be confirmed.

When it is determined that the document has been received (S8: YES), the warehousing flag indicating that the document has been received is associated with the warehousing / delivery identifier, stored in the warehousing / delivery table (S9), and the process is terminated. When it is determined that the document has not been received, that is, it has been issued (S8: NO), the issue flag indicating that the document has been issued is linked to the entry / exit identifier, stored in the entry / exit table (S10), and the imaging process is completed. do. As shown in FIGS. 4 and 5, where the captured cover image contains characters, the characters are recognized as an image (that is, a figure).

After the imaging process is completed, the control unit 21 causes the image processing unit 25 to execute the character image extraction process. FIG. 7 is a flowchart for explaining the character image extraction process by the control device 20, FIG. 8 is an explanatory diagram for explaining contour tracking, and FIG. 9 is an explanatory diagram for explaining a rectangular region.

As shown in FIG. 7, the image processing unit 25 selects one cover image from a plurality of cover images on the front cover or back cover stored in the storage unit 22 (S11). For example, from a plurality of cover images of the front cover, one image in which the entire cover is clearly captured is selected. The image processing unit 25 scans the selected cover image, searches for a start point, and sets it (S12).

As described above, the cover image is a binary image, and as shown in FIG. 8, each pixel of the binary image is given coordinates in the X direction and the Y direction. The pixel includes a black pixel having a brightness equal to or less than a predetermined threshold value and a white pixel having a brightness higher than the threshold value. The black pixel located at the boundary with the white pixel becomes the starting point. Note that the scanning is started from, for example, the upper left pixel of the cover image, scanning is performed along the X direction, and if no black pixel is found, the scanning is performed by moving one pixel in the Y direction and scanning along the X direction. It is said. FIG. 8 shows, for example, a case where a search for a starting point is performed on the cover image of the front cover of FIG. In FIG. 8, the vertex K of the figure “A” is set as the starting point.

For example, as shown by the arrow in FIG. 8, the image processing unit 25 investigates four neighborhoods counterclockwise from the set start point and traces the contour of the figure (S13). A clockwise survey may be performed, or the vicinity of 8 may be surveyed. The image processing unit 25 stores the X-coordinates and Y-coordinates of the black pixels constituting the contour in the storage unit 22 as an array (S14).

The image processing unit 25 creates a rectangular region including the maximum value and the minimum value at the X coordinate and the maximum value and the minimum value at the Y coordinate in the array (S15). For example, in FIG. 8, the K point has the maximum value of the Y coordinate, and the L and M points have the minimum value of the Y coordinate. The M point has the maximum value of the X coordinate, and the L point has the minimum value of the X coordinate. Therefore, the rectangular area is an area in which points K, L, and M as shown in FIG. 9 are located on the sides of the rectangle.

The image processing unit 25 extracts the image in the rectangular area as a character image and stores it in the warehousing / delivery table (S16). The image processing unit 25 determines whether or not scanning has been performed on the entire area of the selected cover image (S17). For example, it is determined whether or not scanning has been performed for all Y coordinates. When scanning is not performed on the entire area (S17: NO), the image processing unit 25 returns the processing to S12 and searches for the start point of the figure in the area excluding the rectangular area.

When scanning is performed on the entire area (S17: YES), it is determined whether or not the contour tracking is completed for both the front and back cover images (S18). If contour tracking has not been completed for both cover images (S18: NO), the process is returned to S11, and one cover image of the front cover (front cover or back cover) for which contour tracking has not been performed is selected. When the contour tracking is completed for both cover images (S18: YES), the image processing unit 25 ends the processing.

After the character image extraction process is completed, the control unit 21 causes the calculation unit 23 to execute the data calculation process. 10 is a flowchart illustrating data calculation processing by the control device 20, FIG. 11 is a conceptual diagram showing an example of an extracted character image, FIG. 12 is a conceptual diagram showing an example of size classification, and FIG. 13 is a diagram. It is a conceptual diagram which shows an example of the coordinate classification of a row with respect to the cover image of the front cover of 4. As shown in FIG. 10, the calculation unit 23 classifies the extracted characters (character images) by size (S31). For example, the calculation unit 23 acquires the X-direction dimension and the Y-direction dimension of each extracted character image (see FIG. 11) and classifies the sizes (see FIG. 12). 11 shows character images A to D, E to I, P to T extracted from the front cover image shown in FIG. 4, and character images X to X extracted from the back cover image shown in FIG. Indicates Z.

The characters on the documents are often written in standardized fonts. Therefore, when the character images are classified based on the size (X-direction dimension and Y-direction dimension), a plurality of characters can be classified into the same group. For example, as shown in FIG. 12, it can be classified into a large size group G1, a medium size group G2, and a small size group G3. In the case of the character images shown in FIG. 11, the character images A to D are classified into the large size group G1, the character images E to I and X to Z are classified into the medium size group G2, and the character images P to T are small. It is classified into size group G3.

The calculation unit 23 counts characters for each classification (S32). For example, in the case of the character image shown in FIG. 11, the number of characters in the large size group G1 is 4, the number of characters in the medium size group G2 is 8, and the number of characters in the small size group G3 is 5. Each number of characters is stored in the warehousing / delivery table (see FIG. 14). The number of characters may be divided into a front cover and a back cover and stored in the warehousing / delivery table.

Next, the calculation unit 23 acquires the coordinates of each character image (S33), and acquires the coordinates of the line composed of the character images (S34). For example, the cover image of the front cover of FIG. 4 includes character images A to D, E to I, and P to T. As shown in FIG. 13, the character images A to D belong to the large size group G1 and have the same Y coordinate y1. The character images E to I belong to the medium size group G2, and the Y coordinates are the same y2. The character images P to T belong to the small size group G3, and the Y coordinates are the same y3. Character image groups that belong to the same size group and include the same Y coordinate are considered to form a traverse. It is considered that the character image group belonging to the same size group and including the same X coordinate constitutes a vertical line.

That is, in S34, the calculation unit 23 acquires the Y coordinate of the character image group that belongs to the same size group and has the same Y coordinate as the horizontal coordinate. The acquired traverse coordinates are divided into a front cover and a back cover and stored in the warehousing / delivery table (see FIG. 14). Further, the X coordinate of the character image group belonging to the same size group and including the X coordinate is acquired as the vertical coordinate. The acquired vertical coordinates are divided into a front cover and a back cover and stored in the warehousing / delivery table.

The calculation unit 23 acquires the number of acquired horizontal row coordinates as the number of horizontal rows, acquires the number of vertical row coordinates as the number of vertical rows (S35), and ends the process. For example, the number of horizontal rows of the front cover of FIG. 4 is 3, and the number of vertical rows is 0. The number of horizontal lines on the back cover of FIG. 5 is 1, and the number of vertical lines is 0. The acquired number of horizontal and vertical rows is stored in the warehousing / delivery table for each of the front and back covers.

After the data calculation process is completed, the control unit 21 executes the update process of the registration table, which will be described later. FIG. 14 is a conceptual diagram showing an example of the warehousing / delivery table stored in the storage unit 22. The entry / exit table stores information regarding documents newly received in the storage 1 and documents newly issued from the storage 1.

FIG. 15 is a conceptual diagram showing an example of a registration table stored in the storage unit 22. The registration table is a table for the user to access when confirming the location of the document. Here, it is assumed that the information of the new document stored in the warehousing / delivery table is not reflected in the registration table. As shown in FIG. 15, the registration table stores the same information as the warehousing / delivery table except that the registration identifier and the thumbnail image are stored in place of the warehousing / delivery identifier and the cover image of the warehousing / delivery table. ing. The control unit 21 refers to the warehousing / delivery table every predetermined time, and if there is information in the warehousing / delivery table, executes the following update process.

FIG. 16 is a flowchart illustrating an update process by the control device 20. The control unit 21 acquires one of the information in the warehousing / delivery table (S41). For example, it is acquired in the order of the entry / exit identifier. In the case of FIG. 14, the information of the warehousing / delivery identifier 1 is first acquired, and each time the update process is repeated thereafter, the warehousing / delivery identifiers 2, 3, ... Are acquired in this order. Next, the control unit 21 determines whether or not the acquired warehousing / delivery table information includes the warehousing flag (S42).

When the warehousing flag is provided (S42: YES), the control unit 21 sequentially acquires the information including the warehousing flag in the registration table (S43). The control unit 21 determines whether or not the acquired warehousing / delivery table information matches the acquired registration table (S44).

For example, the control unit 21 obtains the root mean square error for the number of character images, the number of vertical lines, the number of horizontal lines, the vertical line coordinates, and the horizontal line coordinates, and obtains the consistency between the two. If the error is within the predetermined range, it is determined that they match, and if it is outside the predetermined range, it is determined that they do not match. An error other than the root mean square error may be obtained to evaluate the consistency. For example, a difference may be obtained for each of the above items, and an absolute average error may be obtained from the difference, or a root mean square error or an absolute average error may be obtained for a part of the above items.

When both information match (S44: YES), the control unit 21 changes the issue flag in the acquired registration table information to the receipt flag (S45) and stores the storage position (S46). For example, when the information of the warehousing / delivery identifier 1 and the information of the registration identifier 2 match, the warehousing flag of the registration identifier 2 is changed to the warehousing flag, and the warehousing position of the acquired warehousing / delivery table is stored.

If the two information do not match (S44: NO), the acquired warehousing / delivery table information is newly registered in the registration table (S47). For example, when the information of the warehousing / delivery identifier 2 is acquired, the number of character images, the number of vertical rows, the number of horizontal rows, the vertical row coordinates, and the horizontal row coordinates of the warehousing / delivery identifier 2 are newly registered in the registration table.

The control unit 21 generates a thumbnail image based on the cover image included in the acquired warehousing / delivery table information, associates it with the newly registered information, and stores it in the registration table (S48). For example, the thumbnail image generation unit 26 acquires a character image (S16, see FIG. 11) extracted from the cover image of the front cover and stored in the warehousing / delivery table, and determines from one character image and one character image. Enumerate two character images located within the distance, arrange them before and after, and repeat this. The context can be distinguished from the X coordinate or the Y coordinate. Then, the character images are rearranged in the correct order to generate thumbnail images. A thumbnail image may be generated based on the front cover image of the back cover.

In S42, when the acquired warehousing / delivery table information does not have a warehousing flag (S42: NO), that is, when the information has a warehousing flag, the control unit 21 sequentially orders the information having the warehousing flag in the registration table. Acquire (S49). The control unit 21 determines whether or not the acquired warehousing / delivery table information matches the acquired registration table (S50). When both match (S50: YES), the control unit 21 changes the warehousing flag in the acquired registration table information to the warehousing flag (S51), and deletes the position in the vault. (S52).

In step S50, when the two do not match (S50: NO), the control unit 21 registers the acquired warehousing / delivery table information in the registration table as an error (S53). For example, an error flag is set and the acquired warehousing / delivery table information is registered in the registration table. In FIG. 14, 1 in the "error" column indicates that the error flag is set, and 0 indicates that the error flag is not set. The control unit 21 generates a thumbnail image based on the cover image included in the acquired warehousing / delivery table information, associates it with the error-registered information, and stores it in the registration table (S54).

After performing the processing of S46, S48, S52 or S54, the control unit 21 determines whether or not the information has been confirmed for all the warehousing / delivery identifiers of the warehousing / delivery table (S55). If the information has not been confirmed for all the warehousing / delivery identifiers (S55: NO), the processing is also returned to S41 to acquire the information of the next warehousing / delivery identifier.

When the information is confirmed for all the warehousing / delivery identifiers (S55: YES), the control unit 21 clears the information stored in the warehousing / delivery table (S56), and ends the process. In addition, instead of clearing the information stored in the warehousing / delivery table, an updated flag may be set so that the information can be distinguished from the unupdated information.

In the storage device according to the first embodiment, the camera 9 is arranged inside the storage 1 and automatically identified based on the cover image captured by the camera 9. Further, the position detector 10 acquires the storage position in the storage 1.
FIG. 17 is a schematic view showing an example of a display screen for displaying document information. Since the warehousing position and the warehousing flag are linked to the document identification information and registered, for example, the personal computer, smartphone or tablet terminal is connected to the network by the operation of the user, and the server 40 is requested to request the information of the registration table. The thumbnail image 51 and the position 52 in the storage 1 can be acquired and displayed on the display screen 50. In addition, the user can confirm the information registered as an error and operate the touch panel, keyboard, or the like to correct the information. The user can, for example, operate an icon displayed on the screen to call a pre-installed application program, display information on the above-mentioned documents on the display screen 50, and correct the information. The user can recognize the warehousing or warehousing of documents and the position of the documents in the warehousing 1 without performing the update work by himself / herself at the time of warehousing and warehousing.

The documents stored in the drawer 7 may also be subjected to the above processing. In this case, the camera 9, the lighting, and the position detector 10 are installed on the upper side of the drawer 7, the cover image of the document is imaged, and the position of the drawer 7 in the depth direction is detected.

(Embodiment 2)
Hereinafter, the present invention will be described with reference to the drawings showing the storage device according to the second embodiment. Among the configurations according to the second embodiment, the same configurations as those of the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted. FIG. 18 is a schematic diagram showing a plurality of rectangular areas set for a character having a plurality of figures. In FIG. 18, in order to clarify the rectangular area, the X coordinate, and the Y coordinate, a part of the portion to be displayed in black is displayed in white.

In the second embodiment, a configuration for recognizing a character composed of a plurality of figures as one character image will be described. Here, as shown in FIG. 18, a case where the Japanese "na" is included in the cover page will be described. As shown in FIG. 18, "na" includes a first figure, a second figure, and a third figure.

Similar to the first embodiment, the image processing unit 25 performs contour tracking on each of the first figure, the second figure, and the third figure to create a rectangular area. The image processing unit 25 determines whether or not the two rectangular regions overlap, and if they overlap, integrates the two. Whether or not the two rectangular areas overlap is determined as follows. Let A1 be the point having the largest X coordinate Ax1 and Y coordinate point Ay1 in the rectangular area of the first figure, and let A2 be the point having the smallest X coordinate Ax2 and Y coordinate point Ay2 in the rectangular area of the first figure. Further, the point having the maximum X coordinate Bx1 and the Y coordinate point By1 in the rectangular area of the second figure is referred to as B1, and the point having the minimum X coordinate Bx2 and the Y coordinate point By2 in the rectangular area of the second figure is referred to as B2.

The image processing unit 25 determines whether or not the relationship of Ax1 ≦ Bx1 or Bx2 ≦ Ax2 and Ay1 ≦ By1 or By2 ≦ Ay2 holds, and if so, integrates the first figure and the second figure. That is, a rectangular region is created in which the minimum X coordinate is Ax1, the maximum X coordinate is Bx2, the minimum Y coordinate is By2, and the maximum Y coordinate is Ay1. Then, it is determined whether or not the rectangular area of the third figure also overlaps with the above-mentioned integrated rectangular area. Since the rectangular area of the third figure overlaps the integrated rectangular area, both are integrated.

FIG. 19 is a schematic diagram showing a plurality of circular regions set for a character having a plurality of figures. There is a character such as "ni" in Japanese or "i" or "j" in the alphabet, which is one character composed of a plurality of figures and whose rectangular areas do not overlap. In order to deal with this case, the image processing unit 25 determines whether or not the two rectangular regions overlap by the method shown below, and if they overlap, integrates the two.

In the following description, the integration of the figures related to "ni" shown in FIG. 19 will be described. "Ni" includes the fourth figure, the fifth figure and the sixth figure. The rectangular areas of the fourth figure, the fifth figure, and the sixth figure do not overlap. Focusing on the shape of each rectangular area, it can be seen that the rectangle has a large aspect ratio (long side / short side) extending in one direction.

When the rectangular regions do not overlap, the radius is obtained by the following equation (1), and a circle having the radius is created around the center of the rectangular regions.
Equation (1): Radius = Aspect ratio * √ Area of rectangular area * α
The square root of the area of the rectangular area is used to create a circle whose size is proportional to the size of the figure. Further, α is a coefficient obtained by a trial experiment, and is, for example, 0.636.

Next, when the center of another rectangular area is located within the radius of the circle created around the center of one rectangular area, in other words, the distance between the two rectangular areas can be obtained by the equation (1). If less than the radius, merge the two rectangular areas. Then, the aspect ratio of the integrated rectangular area is obtained, and when the obtained aspect ratio is 1.3 or less, it is considered that the integration has been completed.

For example, when the center of the rectangular area of the fifth figure is located inside the circle created around the center of the rectangular area of the fourth figure, the rectangular areas of the fourth figure and the fifth figure are integrated. Then, when the integrated rectangular area and the rectangular area of the sixth figure overlap, both rectangular areas are integrated. In addition, there is a sufficient distance between the characters, and the figures that make up the adjacent character inside the circle created in the center of the fourth figure, fifth figure, or sixth figure are It shall not be located.

FIG. 20 is a flowchart illustrating a character image extraction process by the control device 20. Since the processes of S11 to S15 and S18 (see FIG. 7) according to the first embodiment are the same in the character image extraction process in the second embodiment, the description thereof will be omitted and only different processes will be described. After the processing of S15, the image processing unit 25 determines whether or not the created rectangular area overlaps with another rectangular area (S61).

When the created rectangular area overlaps with another rectangular area (S61: YES), the image processing unit 25 integrates both rectangular areas (S62) and scans the entire area of the selected cover image. Whether or not it is determined (S63). When scanning is not performed on the entire area (S63: NO), the image processing unit 25 returns the processing to S12 and searches for the start point of the figure in the area excluding the rectangular area created or integrated. When scanning is performed on the entire area (S63: YES), the image in the created or integrated rectangular area is extracted as a character image, stored in the warehousing / delivery table (S64), and the process proceeds to S18.

When the created rectangular area does not overlap with another rectangular area (S61: NO), the image processing unit 25 creates a circle around the center of the created rectangular area (S65), and inside the created circle. , It is determined whether or not the center of the rectangular area of another figure is located (S66).

When the center of the rectangular area of the other figure is located (S66: YES), the image processing unit 25 integrates the rectangular area created by the circle with the rectangular area of the other figure (S67). In S66, when the center of the rectangular region of another figure is not located (S66: NO), or after the processing of S67, the image processing unit 25 proceeds to S63.

(Embodiment 3)
Hereinafter, the present invention will be described with reference to the drawings showing the storage device according to the third embodiment. Of the configurations according to the third embodiment, the same configurations as those of the first or second embodiment are designated by the same reference numerals, and detailed description thereof will be omitted. FIG. 21 is a block diagram illustrating the configuration of the storage device.

The control device 20 of the storage device includes a control unit 21, a storage unit 22, and a timer 24, and is connected to the server via a network. The server includes a control unit 41, a calculation unit 23, an image processing unit 25, a thumbnail image generation unit 26, a storage unit 42, and a cover estimation unit 43. The control device 20 executes the above-mentioned imaging process and transmits the process result to the server. The server stores the received processing result in the storage unit 42, and executes the above-mentioned character image extraction processing and data calculation processing. The warehousing / delivery table and the registration table are stored in the storage unit 42 of the server. The control unit 41, the calculation unit 23, the image processing unit 25, the thumbnail image generation unit 26, the storage unit 42, and the cover estimation unit 43 may be provided in the control device 20.

The cover estimation unit 43 can be composed of, for example, a plurality of CPUs, a plurality of GPUs, a combination of CPUs and GPUs, and a storage unit. An estimation model for estimating the cover image is created by a predetermined programming language, and the estimation model is stored in the storage unit of the cover estimation unit 43.

FIG. 22 is an explanatory diagram schematically showing a configuration example of the cover estimation unit 43. In FIG. 22, the cover estimation unit 43 is configured by a neural network model, and is composed of, for example, a CNN (Convolutional Neural Network). The configuration of the cover estimation unit 43 is not limited to the multi-layer neural network (deep learning) as in the example of FIG. 22, and other machine learning (for example, support vector machine, random forest) algorithm is used. You can also.

As shown in FIG. 22, the cover estimation unit 43 is composed of an input layer, an output layer, and a plurality of intermediate layers. In FIG. 22, two intermediate layers are shown for convenience, but the number of layers of the intermediate layer is not limited to two, and may be three or more. The number of nodes in the input layer is, for example, n, and the number of nodes in the output layer is, for example, 3. The number of nodes in the output layer may be 4 or more or 2 or less.

There are a plurality of nodes (neurons) in the input layer, the output layer, and the intermediate layer, and the nodes in each layer are connected to the nodes in the previous and next layers in one direction with a desired weight. A value equal to the number n of nodes in the input layer is input to the input layer as input data of the cover estimation unit 43.

The data input to each node of the input layer, that is, the number of character images, the number of vertical rows, the number of horizontal rows, the vertical row coordinates and the horizontal rows coordinates, and the warehousing flag or the warehousing flag related to the predetermined cover image stored in the warehousing / delivery table are When given as an input to the first intermediate layer, the intermediate layer calculates the output value using the weight and activation function, the calculated output value is input to the next intermediate layer, and so on. The value is transmitted to the subsequent layer (lower layer) one after another until the output is output. In the estimation model, the number of character images, the number of vertical rows, the number of horizontal rows, the vertical row coordinates, and the horizontal row coordinates are used as training data, and the thumbnail image corresponding to the training data is used as a teacher label in advance by a learning processing unit (not shown). Learning is taking place. The learning algorithm of the learning processing unit calculates all the weights that connect the nodes. The thumbnail image of the teacher label is created by the method shown in the first embodiment.

Based on the input data, the output layer outputs the top three probabilities among the probabilities corresponding to the thumbnail images stored in the registration table. The output with the highest probability is the estimation result. For example, when the probability of being a thumbnail image of registration identifier 1 is 80%, the probability of being a thumbnail image of registration identifier 2 is 10%, and the probability of being a thumbnail image of registration identifier 3 is 5%, the cover estimation unit 43 determines. It is presumed that the cover page related to the input data is the cover page of the registration identifier 1.

When the input data includes the warehousing flag, the cover estimation unit 43 is configured to output the probability corresponding to the thumbnail image in which the warehousing flag is set, and the input data includes the warehousing flag. If so, the cover estimation unit 43 is configured to output the probability corresponding to the thumbnail image for which the warehousing flag is set.

The control unit 41 of the server executes the update process every predetermined time elapses. FIG. 23 is a flowchart illustrating the update process by the server. The control unit 41 of the server determines whether or not there is data in the warehousing / delivery table (S71). If there is no data (S71: NO), the process returns to S71 after waiting for a predetermined time (S78). The server starts the update process every predetermined time elapses.

When there is data in the warehousing / delivery table (S71: YES), the control unit 41 acquires the input data from the warehousing / delivery table (S72), and the cover estimation unit 43 tells the cover estimation unit 43 which registration identifier is the thumbnail image. For example, it is made to estimate which cover page it is (S73). The control unit 41 acquires an estimation result from the cover estimation unit 43 (S74).

The control unit 41 updates the registration table based on the estimation result (S75). When the input data includes the warehousing flag, the control unit 41 changes the warehousing flag of the document with the highest probability to the warehousing flag. If the input data includes an issue flag, change the issue flag of the document with the highest probability to the issue flag. In addition, when the input data includes a warehousing flag and the highest probability of the estimation result is less than or equal to a predetermined probability, for example, 10% or less, it is determined that the document is newly warehousing and registered. Make a new registration in the table. When the input data includes the issue flag and the highest probability of the estimation result is less than or equal to a predetermined probability, for example, 10% or less, it is determined that an error is made and an error is registered in the registration table.

The control unit 41 determines whether or not the update process has been performed on all the data in the warehousing / delivery table (S76). When the update process is not performed for all the data (S76: NO), the control unit 41 returns the process to S72 and acquires the next input data. When the update process is performed for all the data (S76: YES), the control unit 41 clears the warehousing / delivery table (S77), waits for a predetermined time (S78), and then returns the process to S71.

The server 40 may be provided with a pre-trained classification estimation model using the cover image as training data and the "document" or "drawing" corresponding to the learning data as teacher labels. In this case, by inputting the cover image into the input layer of the classification estimation model, the probabilities corresponding to the "document" and the "drawing" are output from the output layer. The control unit 41 of the server 40 adopts a high-probability output, and registers the adopted classification (“document” or “drawing”) in the registration table in association with the registration identifier corresponding to the input cover image. do. When searching for a document, the user can specify "document" or "drawing" to search.

For example, a case of searching a resume as a "document" will be described. Here, search for resumes in the same format. The number, position, and dimensions of a plurality of squares (entry fields) are read in advance from the cover image of the received resume, and are registered in the registration table together with the thumbnail image of the resume. The user operates the external device 30 to select a resume having a desired format from the thumbnail images of the received resumes in the registration table. The external device 30 causes the display screen to display a plurality of thumbnail images showing a resume having the same number, position, and dimensions as the selected resume, that is, a resume similar to the selected resume.

The case of searching for "drawing" will be described. The feature points are read from the cover image of the received drawing in advance, and are registered in the registration table together with the thumbnail image of the drawing. The feature points correspond to, for example, the number, position, and dimensions of the image (graphic image) of the graphic described in the drawing. The graphic image corresponds to the character image described above, and is determined by setting a rectangular area in the same manner as the character image. The user operates the external device 30 to select a thumbnail image similar to the desired drawing from the thumbnail images of the stored drawings in the registration table. The external device 30 causes the display screen to display a plurality of thumbnail images of drawings having the same number, positions, and dimensions of graphic images as the selected drawing, that is, drawings similar to the selected drawing. Instead of the thumbnail image, the cover image may be registered in the registration table so that the cover image is selected. For the determination of the degree of similarity (degree of agreement), for example, the same method as in the first embodiment is adopted (see S44 and S50).

(Embodiment 4)
Hereinafter, the present invention will be described with reference to the drawings showing the storage device according to the fourth embodiment. Of the configurations according to the fourth embodiment, the same configurations as those of the first to third embodiments are designated by the same reference numerals, and detailed description thereof will be omitted. FIG. 24 is a block diagram illustrating the configuration of the storage device.

The storage device according to the fourth embodiment includes the control device 20 and the server as in the third embodiment. Unlike the third embodiment, the server according to the fourth embodiment includes a character estimation unit 44. The character estimation unit 44 can be composed of, for example, a plurality of CPUs, a plurality of GPUs, a combination of CPUs and GPUs, and a storage unit. An estimation model for estimating characters from a character image is created by a predetermined programming language, and the estimation model is stored in a storage unit of the character estimation unit 44. The server configuration may be provided in the control device 20.

FIG. 25 is an explanatory diagram schematically showing a configuration example of the character estimation unit 44. In FIG. 25, the character estimation unit 44 is configured by a neural network model, for example, CNN. The configuration of the character estimation unit 44 is not limited to the multi-layer neural network as in the example of FIG. 25, and other machine learning algorithms can also be used.

As shown in FIG. 25, the character estimation unit 44 includes an input layer, an output layer, and a plurality of intermediate layers. In FIG. 25, two intermediate layers are shown for convenience, but the number of layers of the intermediate layer is not limited to two, and may be three or more. The number of nodes in the input layer is, for example, n, and the number of nodes in the output layer is, for example, N.

There are a plurality of nodes (neurons) in the input layer, the output layer, and the intermediate layer, and the nodes in each layer are connected to the nodes in the previous and next layers in one direction with a desired weight. A value equal to the number n of nodes in the input layer is input to the input layer as input data of the character estimation unit 44.

The control unit 41 of the server causes the input layer to input the brightness of the pixels of the character image extracted by the image processing unit 25. At this time, the control unit 41 unifies the pixels of the large, medium, and small size character images into a predetermined dimension in the image processing unit 25, converts them into, for example, a 16 × 16 pixel image, and then each pixel. The brightness of is input to the input layer.

The data input to each node of the input layer, that is, the brightness of the n (for example, 256) pixels constituting the character image, is given to the first intermediate layer as an input, and the intermediate layer is used with a weight and an activation function. Calculates the output value, inputs the calculated output value to the next intermediate layer, and transmits the value to the subsequent layer (lower layer) one after another until the output layer outputs the output value in the same manner. In the estimation model, the brightness of the pixels constituting the character image is used as learning data, and the character corresponding to the learning data (for example, a character code that can be used by a computer) is used as a teacher label by a learning processing unit (not shown). , Learning has been done in advance. The learning algorithm of the learning processing unit calculates all the weights that connect the nodes.

The output layer outputs the probability of being a character for each of the N characters based on the input data. For example, the probability P [A] of the character A, the probability P [B] of the character B, and the like are output. The output with the highest probability is the estimation result. For example, when the probability of being "A" is 98%, the probability of being "B" is 1%, and the probability of being "D" is 0.5%, the character estimation unit 44 determines that the character related to the input data is ". It is estimated to be "A". The output layer is not limited to the alphabet, but can also output the probability that it is a character of another original language such as hiragana, katakana, or kanji.

The storage unit 42 of the server stores the warehousing / delivery table. FIG. 26 is a conceptual diagram showing an example of the warehousing / delivery table. Similar to the first embodiment, the warehousing / delivery table stores the warehousing / delivery identifier, the cover images of the front and back covers, the warehousing flag or the warehousing flag, and the position in the warehousing 1.

The storage unit 42 of the server stores the character table associated with the warehousing / delivery identifier. FIG. 27 is a conceptual diagram showing an example of a character table. In FIG. 27, the character table of the warehousing / delivery identifier 1 is shown. The character table stores the extracted character image, the characters corresponding to the character image, and the coordinates of the character image. The character is a character that has the highest probability by the character estimation unit 44. The coordinates are the coordinates of any point in the character image. For example, the coordinates of the center point or the apex of the rectangular region created in the first embodiment can be mentioned. A character table is created in the same manner for the warehousing / delivery identifier 2 and thereafter.

The storage unit 42 of the server stores the registration table. FIG. 28 is a conceptual diagram showing an example of the registration table. Similar to the first embodiment, the registration table stores a registration identifier, a thumbnail image, a warehousing flag or a warehousing flag, and a position in the warehousing 1.

The storage unit 42 of the server stores the character table associated with the registration identifier. FIG. 29 is a conceptual diagram showing an example of a character table. In FIG. 29, the character table of the registration identifier 1 is shown. The character table stores the extracted character image, the characters corresponding to the character image (in other words, the character code that can be used by a computer), and the coordinates of the character image. A character table is created in the same manner for the registration identifier 2 and later.

FIG. 30 is an explanatory diagram schematically showing a configuration example of the cover estimation unit 43. In FIG. 30, the cover estimation unit 43 is configured by a neural network model, and is composed of, for example, CNN. Other machine learning algorithms can be used for the configuration of the cover estimation unit 43.

As shown in FIG. 30, the cover estimation unit 43 is composed of an input layer, an output layer, and a plurality of intermediate layers. In FIG. 30, two intermediate layers are shown for convenience, but the number of layers of the intermediate layer is not limited to two, and may be three or more. The number of nodes in the input layer is, for example, n, and the number of nodes in the output layer is, for example, 3. The number of nodes in the output layer may be 4 or more or 2 or less. The warehousing or warehousing flag and the characters and coordinates stored in the character table corresponding to the warehousing / delivery identifier are input to each node of the input layer. Characters and coordinates are input as linked sets.

When the data input to each node of the input layer is given as an input to the first intermediate layer, the intermediate layer calculates the output value using the weight and activation function, and the calculated output value is transferred to the next intermediate layer. The value is transmitted to the subsequent layers (lower layers) one after another until the input is input and the output layer outputs the output value in the same manner. In the estimation model, characters and coordinates are used as learning data, and thumbnail images corresponding to the learning data are used as teacher labels, and learning is performed in advance by a learning processing unit (not shown). The learning algorithm of the learning processing unit calculates all the weights that connect the nodes. The thumbnail image of the teacher label is created by the method shown in the first embodiment. Based on the input data, the output layer outputs the top three probabilities among the probabilities corresponding to the thumbnail images stored in the registration table. The output with the highest probability is the estimation result.

When the input data includes the warehousing flag, the cover estimation unit 43 is configured to output the probability corresponding to the thumbnail image in which the warehousing flag is set, and the input data includes the warehousing flag. If so, the cover estimation unit 43 is configured to output the probability corresponding to the thumbnail image for which the warehousing flag is set.

The control unit 41 of the server executes the update process every predetermined time elapses. Since the update process is the same as the update process of the third embodiment except that the input data is the characters and coordinates stored in the character table corresponding to the warehousing or warehousing flag and the warehousing / delivery identifier (FIG. 23). (See), the detailed description thereof will be omitted. The input data may include the X-direction size and the Y-direction size of the character image.

FIG. 31 is a schematic diagram showing an example of a display screen for displaying document information. The external device 30 (terminal) is connected to the network by the operation of the user, and requests and acquires the information of the registration table from the server 40. The terminal accepts character input and searches for documents. On the display screen 50 of the terminal, for example, the thumbnail image 51 of the document, the position 52 in the storage 1, the search character 53, and the match probability 54 between the search character and the candidate document are displayed as search results in descending order of probability. NS. The probability is the number of matching characters between the search character 53 and the characters listed on the cover of the candidate document, and the ratio of the lengths of consecutive matching characters (for example, the search character is ABCDE and ABCD is continuous). If they match, the ratio of the lengths of the matching characters is 0.8). The application program installed on the terminal executes the above display and search process. The character estimation unit 44 may convert a character image into characters by using OCR (Optical Character Recognition) instead of the neural network model. In the third and fourth embodiments, as in the second embodiment, the processing for integrating the rectangular regions may be performed to extract the character image.

When a plurality of storages 1 are used, an identifier may be assigned to each storage 1 and stored in the registration table. FIG. 32 is a schematic view showing an example of a display screen for displaying information on documents stored in a plurality of storages 1. As shown in FIG. 32, the display screen 50 of the terminal stores the thumbnail image 51 of the document, the position 52 in the storage 1, the search character 53, the probability of matching the search character with the candidate document 54, and the candidate document. Information 55 indicating the storage is displayed in descending order of probability. A storage 1 in which desired documents are stored can be displayed, and convenience can be enhanced in a facility (for example, a library) that requires a large number of storages 1. For example, the user operates an icon displayed on the screen to call a pre-installed application program, display information on the above-mentioned documents (see FIGS. 31 and 32) on the display screen 50, and correct the information. can do.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The technical features described in each example can be combined with each other and the scope of the invention is intended to include all modifications within the claims and scope equivalent to the claims. Will be done.

1 Storage 9 Camera 10 Position detector 20 Control device 23 Calculation unit 25 Image processing unit 40 Server 43 Cover estimation unit 44 Character estimation unit

Claims (7)

A storage for storing documents and
An imaging unit that is arranged on the inner surface of the storage and captures the cover of the document to be stored or delivered.
A generation unit that generates a feature amount of the document to be stored based on the cover image captured by the imaging unit, and a generation unit.
A storage device including a registration unit that registers the feature amount of the document generated and stored in the generation unit in association with a warehousing flag indicating warehousing. A position detector for detecting the storage position of the document in the storage is provided.
The storage device according to claim 1, wherein the registration unit registers the feature amount of the document to be stored in association with the storage position detected by the position detector. A second generation unit that generates a feature amount of the document to be delivered based on the cover image captured by the imaging unit, and a second generation unit.
An update unit that corresponds to the feature amount generated by the second generation unit, erases the warehousing position associated with the feature amount registered in the registration unit, and rewrites the warehousing flag to the warehousing flag indicating the warehousing. 2. The storage device according to claim 2. The generation unit has an extraction unit that extracts a character image from the cover image, and any of claims 1 to 3 that generates a feature amount of the document based on the character image extracted by the extraction unit. The storage device described in one. The extraction unit
A tracking unit that tracks the contours of a plurality of figures included in the character image, and
A creation unit that creates an area surrounding each of the plurality of figures based on the contour tracked by the tracking unit, and a creation unit.
A determination unit that determines whether or not the areas created by the creation unit overlap, and a determination unit.
The storage device according to claim 4, further comprising an integrated unit that integrates the overlapping regions when the determination unit determines that they overlap. When the figure includes the first figure and the second figure, in the first direction, one end of the second figure is arranged between both ends of the first figure, and the determination unit is in the first direction. The storage device according to claim 5, wherein when one end of the second figure is arranged between both ends of the first figure in the intersecting second direction, it is determined that the first figure and the second figure overlap. .. The creation part
A rectangle creation part that creates a rectangle area, and
A derivation unit for deriving the aspect ratio of the rectangular region created by the rectangle creation unit, and a derivation unit.
It has a circle creating unit that creates a circle with a radius determined based on the aspect ratio derived by the deriving unit around the center of the rectangular region.
The storage device according to claim 5, wherein the determination unit determines whether or not the circles created by the circle creation unit overlap when the rectangle regions created by the rectangle creation unit do not overlap.

Images