JP5169596B2 - Information management apparatus, information management method, and program - Google Patents

Description

  The present invention relates to an information management apparatus, an information management method, and a program.

  In recent years, while documents and the like are generally created and managed electronically, handwritten documents still exist. In particular, handwriting is convenient when creating a simple memo or the like.

  Incidentally, there may be a relationship between an electronic document and a handwritten document. For example, when a meeting is being conducted with reference to an image in which an electronic file is projected by a projector or a distribution material printed based on the electronic file, a memo or the like is written by hand on paper such as a notebook There is. In this case, it can be said that the projected electronic document or the electronic document of the distribution source and the memo handwritten during the meeting are related.

Therefore, conventionally, techniques for associating and managing related electronic documents and handwritten documents have been studied (for example, Patent Document 1 and Patent Document 2).
JP 2004-080601 A JP 2007-048217 A JP 2006-260522 A

  However, with the conventional technology, for example, it is difficult to associate and manage a handwritten memo made on a notebook or notebook and the electronic document. In addition, various restrictions exist such that it is necessary to perform handwriting on a paper on which a barcode is printed for association.

  The present invention has been made in view of the above points, and is an information management apparatus and an information management method capable of appropriately associating handwritten information created in connection with an operation of an electronic file and the electronic file. And to provide a program.

  Accordingly, in order to solve the above-described problem, the present invention provides an information management apparatus having an information storage unit and an information search unit, wherein the information storage unit identifies the electronic data according to the occurrence of an operation on the electronic data. And an operation timing recording means for associating the operation timing with the timing information indicating the occurrence timing of the operation, and a first image for acquiring, from the image input device, image data obtained by capturing at least a part of a handwritten paper document A feature quantity that is recorded in the storage device in association with data acquisition means, first feature quantity extraction means for extracting the feature quantity of the image data, timing information indicating the acquisition timing of the image data, and the feature quantity Recording means, wherein the information search means is a second feature quantity extraction means for extracting feature quantities of image data input from the image input device, and features that match the extracted feature quantities Search means from the storage device, and the identification information associated with the time information related to a time within a predetermined period from the time indicated by the time information associated with the searched feature amount It has a related information acquisition means acquired from an apparatus, and a display control means for displaying information based on the acquired identification information.

  In such an information management apparatus, handwritten information created in association with the operation of the electronic file can be appropriately associated with the electronic file.

  According to the present invention, it is possible to provide an information management apparatus, an information management method, and a program capable of appropriately associating handwritten information created in connection with an operation of an electronic file and the electronic file.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a hardware configuration example of an information management apparatus according to an embodiment of the present invention. In FIG. 1, an information management apparatus 10 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, and the like that are mutually connected by a bus B.

  A program for realizing processing in the information management apparatus 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 on which the program is recorded is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program need not be installed from the recording medium 101 and may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program and also stores necessary files and data.

  The memory device 103 reads the program from the auxiliary storage device 102 and stores it when there is an instruction to start the program. The CPU 104 realizes functions related to the information management apparatus 10 according to a program stored in the memory device 103.

  The information management apparatus 10 can be connected to the display device 106, the input device 107, the projection device 108, the image input device 109, and the like via a predetermined hardware interface (for example, a USB (Universal Serial Bus) connector). . The display device 106 is, for example, a CRT display or a liquid crystal display, and displays a GUI (Graphical User Interface) or the like by a program. The input device 107 includes a keyboard and a mouse, and is used for inputting various operation instructions. The projection device 108 is a so-called projector, and projects an image to be projected (for example, an image displayed on the display device 106). The image input device 109 is, for example, a scanner, a digital camera, a mobile phone with a camera, or the like, and takes an image of a subject. Note that the projection device 108 and the image input device 109 do not need to be always connected to the information management device 10, and may be connected when necessary (when an image is projected or when an image is input).

  FIG. 2 is a diagram illustrating a software configuration example of the information management apparatus according to the embodiment of the present invention. In FIG. 1, the information management apparatus 10 includes an information storage unit 11, an information search unit 12, an event database 13, an association database 14, an OS (Operating System) 15, and the like.

  The information storage unit 11 performs the electronic file or the image data (hereinafter referred to as both) in response to an operation on an electronic file (electronic document) stored in the auxiliary storage device 102, an input of image data from the image input device 109, or the like. Information is stored (recorded) in the event database 13 and the association database 14. In the figure, the information storage unit 11 includes an event acquisition unit 111, a rasterization unit 112, a feature amount extraction unit 113, an image acquisition unit 114, an image synthesis unit 115, an information recording unit 116, a relation generation unit 117, and the like.

  On the other hand, in response to an input of a search request for information related to image data input from the image input device 109, the information search unit 12 relates to the image data from information related to electronic data stored in the event database 13. Search the information you want. In the figure, the information search unit 12 includes an image input unit 121, a frame selection unit 122, a feature amount extraction unit 123, a feature amount comparison unit 124, a related information search unit 125, a search result display unit 126, and the like.

  The event database 13 is a database in which information related to electronic data is stored by the information storage unit 11, and is constructed (recorded) in the auxiliary storage device 102. The association database 14 is a database that holds association information of information recorded in the event database 13, and is constructed (recorded) in the auxiliary storage device 102.

  FIG. 2 shows an example in which the information storage unit 11 and the information search unit 12 are included in the same device (information management device 10), but each is separately connected via a network. It may be mounted on the device. That is, the information management apparatus 10 in the present embodiment also represents a system composed of a plurality of apparatuses (computers).

  Hereinafter, the processing procedure of the information management apparatus 10 will be described separately for information storage and information search.

  FIG. 3 is a flowchart for explaining the processing procedure of the information management apparatus during information accumulation.

  The event acquisition unit 111 constantly monitors operation input by the operator (event notification from the OS 15 based on the operation input) (S101, S102). When the event acquisition unit 111 detects an operation input (event) by the operator (Yes in S102), the event acquisition unit 111 branches the process according to the event type.

  For example, when an electronic file saved in the auxiliary storage device 102 is opened by an application program (not shown), the event acquisition unit 111 detects an event indicating that the electronic file has been opened (opened). In this case (Yes in S103), the event acquisition unit 111 acquires the file name of the electronic file, the occurrence date and time of the event (electronic file open), and the contents of the electronic file (S104). In addition, the event acquisition unit 111 determines the time zone to which the event occurrence date / time belongs. Here, the time zone refers to information indicating a period defined by dividing a predetermined unit into 24 hours in a day, for example. For example, when the predetermined unit is 2 hours, a time period of 1 to 12 is defined. Subsequently, the information recording unit 116 registers (records) the information acquired by the event acquisition unit 111 in the event database 13 as one record (entry) (S105).

  FIG. 4 is a diagram illustrating a configuration example of the event database. In the figure, the event database 13 is configured to manage an event ID, a file name, an event type, an occurrence date and time, a time zone, a feature amount, and a raster image for each event. The event ID is a unique ID for each event (each record (row) of the event database 13) and is assigned by the information recording unit 116. The file name is the name of the file that is the subject of the event, and the value acquired by the event acquisition unit 111 is registered. The event type is a type of event, and corresponds to file open (“open”) here. The occurrence date / time and time zone are the event occurrence date / time and the time zone to which the occurrence date / time belongs, and the value acquired by the event acquisition unit 111 is registered. The feature amount and the raster image will be described later. The records in the event database 13 are hereinafter referred to as “event records”.

  The same processing is executed when the opened electronic file is closed (S111 to S113). Therefore, in this case, an event record whose event type is “close” is registered in the event database 13.

  When printing is performed for an electronic file, the event acquisition unit 111 detects an event indicating that the electronic file has been printed. In this case (Yes in S121), the event acquisition unit 111 acquires the file name of the printed electronic file, the occurrence date and time of the event (electronic file printing), the contents of the electronic file, the print page, and the like (S122). In addition, the event acquisition unit 111 determines the time zone to which the event occurrence date / time belongs. Subsequently, the rasterization unit 112 converts data relating to the print page in the electronic file into image data (raster image) (S123). Subsequently, the feature amount extraction unit 113 extracts (generates) the feature amount of the image data generated by the rasterization unit 112 (S124). Details of the feature amount will be described later.

  Subsequently, the information recording unit 116 additionally registers the information acquired by the event acquisition unit 111 in the event database 13 as one event record, similarly to step S105 or S113 (S125). However, in this case (in the case of printing), the image data (raster image) generated by the rasterizing unit 112 and the feature amount of the image data are also registered in the same event record.

  When the projection by the projection device 108 is started (that is, when an image is output to the projection device 108), the event acquisition unit 111 detects an event indicating the start of projection (output of the image to the projector). . In this case (Yes in S131), the event acquisition unit 111 includes the file name and event (video) of an electronic file opened by an active (targeted operation) application included in the video output to the projection device 108. The date and time of occurrence of the output and the contents of the electronic file are acquired (S132). In addition, the event acquisition unit 111 determines the time zone to which the event occurrence date / time belongs. Subsequently, the rasterizing unit 112 converts the data related to the page currently displayed (projected) in the electronic file into image data (raster image) (S133). Subsequently, the feature amount extraction unit 113 extracts (generates) the feature amount of the image data generated by the rasterization unit 112 (S134). Subsequently, as in step S125, the information recording unit 116 additionally registers the information, image data, and feature amount acquired by the event acquisition unit 111 in the event database 13 as one event record (S135).

  Furthermore, when image data of a paper document is captured from the image input device 109 and the image data is input to the information management device 10, the event acquisition unit 111 detects an event indicating input of the image data. In this case (Yes in S141), the image acquisition unit 114 acquires input image data and records it in the memory device 103. Note that image data input from the image input device 109 is not limited to wired communication, and wireless communication such as Bluetooth or infrared may be used. The positioning as the image input device 109 may be a recording medium (memory card or the like) on which image data is recorded, an external computer that manages the image data, or the like.

  Incidentally, one or more still image data or moving image data can be input from the image input device 109. Moving image data can be input for a stationary subject such as a paper document when the camera is moved while the part of the paper document is continuously imaged to capture the entire surface or part of the paper document. is there. By doing so, a high-resolution image can be obtained even when the resolution of the camera is low.

  When the image data acquired by the image acquisition unit 114 is still image data (Yes in S142), the image acquisition unit 114 saves (records) the image data recorded in the memory device 103 as a file in the auxiliary storage device 102. (S143). Therefore, a file name is assigned to the image data automatically or based on an input from the operator. Further, the event acquisition unit 111 acquires the file name of the image data and the date and time of occurrence of the event (input of image data). Furthermore, the event acquisition unit 111 determines the time zone to which the event occurrence date belongs.

  Subsequently, the feature amount extraction unit 113 extracts (generates) the feature amount of the image data stored in the file (S144). Subsequently, the information recording unit 116 additionally registers the information acquired by the event acquisition unit 111, the image data, and the feature amount in the event database 13 as one event record (S145).

  On the other hand, when the image data acquired by the image acquisition unit 114 is moving image data (No in S142), the image composition unit 115 generates one piece of still image data from the moving image data (S151). The still image data is image data for an area (for example, the entire surface) where the camera is moved on the paper document. In order to create one piece of large still image data from moving image data, a technique described in Japanese Patent Application Laid-Open No. 2004-334843 and the like that generates a large image by combining a plurality of partial images may be used.

  Subsequently, the image composition unit 115 stores the generated image data (still image data) as a file in the auxiliary storage device 102 (S152). Further, the event acquisition unit 111 acquires the file name of the image data and the date and time of occurrence of the event (input of image data). Furthermore, the event acquisition unit 111 determines the time zone to which the event occurrence date belongs. Subsequently, the feature amount extraction unit 113 extracts (generates) the feature amount of the image data stored in the file (S153). Subsequently, the information recording unit 116 additionally registers the information, image data, and feature amount acquired by the event acquisition unit 111 as one event record in the event database 13 (S154).

  Subsequent to step S145 or step S154, the association generation unit 117 performs another event record (hereinafter referred to as “key event record”) registered in response to the input of the current image data (hereinafter referred to as “key event record”). , "Related event record") is searched (S155). Details of this step will be described.

  FIG. 5 is a flowchart for explaining the related event record search processing by the relation generation unit.

  First, an event record having a feature amount that matches the feature amount of the key event record is searched (S201). If there is a searched event record (Yes in S202), the event record is set as a related event record (S203). . Note that the match may include not only a perfect match but also a state in which the feature amount is similar or approximate. The determination of similarity or approximation differs depending on the characteristics or configuration of the feature amount, but may be performed based on a threshold value regarding the feature amount to be adopted.

  Subsequently, an event record related to a time zone that is the same as or close to the time zone of the key event record is searched (S204). If there is a searched event record (Yes in S205), the event record is set as a related event record ( S206). The close time zone may be determined based on a threshold value set in advance in the auxiliary storage device 102. For example, when the threshold value for determining the near time zone is “2”, the event record relating to the same time zone and the previous time zone is set as the related event record.

  That is, the relationship between the key event record and the related event record means that the feature amounts match or the time zone is close (within a predetermined range). A plurality of related event records can be searched.

  Returning to FIG. When the related event record is searched, the relationship generation unit 117 registers a record indicating the association information between the key event record and the related event record (hereinafter referred to as “association record”) in the association database 14 (S156). .

  FIG. 6 is a diagram illustrating a configuration example of the association database. In the figure, the association database 14 is configured to be able to manage relation ID, key event ID, number of associations, ID string, association type, date of occurrence, and time zone for each association information (association record). The relation ID is a unique ID for each association information. The key event ID is the event ID of the key event record. The number of associations is the number of event records that are determined to be associated based on the key event records. In the example of the figure, the key event records themselves are also counted. The related ID column is a set of IDs of related event records, and in the example of FIG. The association type is information indicating whether the association is associated based on the feature amount or the time. When the association is performed based on the feature amount, the event type of the related event record is registered. That is, when a plurality of related event records for one key event record are classified into a plurality of types of event types, a plurality of association records are registered for each event type. The occurrence date is the occurrence date of the event related to the key event record. The time zone is a time zone to which the associated event record belongs.

  For example, when a paper document with an image input relates to an electronic file that has already been printed, the input image data is associated with the original electronic file based on the matching of the feature amounts. (Case A). Further, the input image data is associated with an electronic file operated (opened, closed, printed, projected, etc.) in a time zone close to the input time zone of the image data (Case B).

  In case A, a handwritten memo made on a paper document that is a printed matter of an electronic file can be associated with the electronic file. In case B, the electronic file can be associated with a handwritten memo made on a sheet other than the printed matter of the electronic file, such as a notebook or a notebook during the operation of the electronic file.

  Note that an event corresponding to each operation is detected when an interrupt is generated from the input device 106, the projection device 108, the image input device 109, or the like to the OS 15. That is, the event acquisition unit 111 detects each event by monitoring the occurrence of an interrupt to the OS 15.

  For example, when the operator inputs to the input device 107, data corresponding to the input is input from the input device 107 to the information management device 10, and based on the data, an image is displayed on the display device 106. Output is done. Event information corresponding to these inputs and outputs can be acquired from the OS 15. Therefore, the event acquisition unit 111 can determine the event type by referring to such event information.

  Incidentally, the event database 13 in FIG. 4 corresponds to the following specific example. First, aaa. The ppt will be opened on February 1, 2008 at 13:55. aaa. After editing the ppt, printing is performed at 14:20 and the file is closed. Note that aaa. The printed matter of ppt is used as handout material for the conference. Subsequently, at the meeting from 15:00, aaa. The ppt is opened at 15: 2 and projected by the projector 108. During the meeting, aaa. Memo writing is performed on printed materials (distributed materials) of ppt and paper such as notebooks. The meeting ends and at 16: 4 aaa. ppt is closed. Thereafter, image data (hereinafter referred to as “image data A”) on which paper was written during the meeting is input from the image input device 109 (scanner, digital camera, or the like), and still1. Saved as jpg. Also, aaa. Image data (hereinafter referred to as “image data B”) of which a memo has been written on a printed material (distributed material) of ppt is input from the image input device 109 in a moving image format, and one large size is obtained by image synthesis. Is stored as a still image data file (combined1.jpg).

  When image data A is input (when an event record with event ID = 7 is created), a feature amount search is performed. Since the image data A is obtained by imaging a sheet different from the distribution material, the related event record is not searched in the feature amount search. Subsequently, a time zone search is performed, and event records with event IDs 4 to 7 are searched as related event records. The search result is registered in the association database 14 as an association record with relation ID = 1.

  A feature amount search is also performed when image data B is input (when an event record with event ID = 7 is created). The image data B is obtained by writing a note on the distribution material (printed material of aaa.ppt), which is input in a moving image format and synthesized into still image data. Therefore, aaa. Two event records having event ID = 2 and event ID = 5 having a feature quantity of ppt are searched as related event records. Since the event types of the two related event records are different, an association record with relation ID = 2 and relation ID = 3 is registered for each related event record. Subsequently, a time zone search is performed, and event records with event IDs 4 to 7 are searched as related event records. The search result is registered in the association database 14 as an association record with relation ID = 4.

  Next, information retrieval will be described. For example, a few days later, you have a notebook or notebook with handwritten notes in a meeting, or paper documents such as handouts for meetings, and you want to search for information (such as electronic files) related to the paper documents. .

  FIG. 7 is a flowchart for explaining the processing procedure of the information management apparatus during information retrieval.

  In step S <b> 301, when a notebook, a handwritten memo of a notebook, or a conference distribution material is captured by the image input device 109, the image data input unit 121 inputs the captured image data (still image data or moving image data). The information is input (captured) from the device 109 to the information management device 10 and recorded in the memory device 103. When the input image is moving image data (No in S302), the frame selection unit 122 selects one frame in the image sequence (frame) of the moving image data, and extracts the feature amount of the image data related to the selected frame. The data is input to the unit 123 (S303). The frame selection by the frame selection unit 122 is performed by, for example, performing edge strength extraction for each frame and determining whether the result satisfies a predetermined condition (for example, the sum of differential values of each pixel is equal to or greater than a predetermined value). Just do it. In this case, a frame that satisfies the predetermined condition is selected.

  On the other hand, when the input image is still image data (Yes in S302), the still image data is input to the feature amount extraction unit 123 as it is.

  Subsequently, the feature amount extraction unit 123 extracts (generates) the feature amount of the image data input in (1) by the same method as the method by the feature amount extraction unit 113 of the information storage unit 11 (S304). The feature amount extracted here is hereinafter referred to as “current feature amount”. Subsequently, the feature amount comparison unit 124 compares the current feature amount with the feature amount registered in the event database 13 and determines an event record related to the feature amount that matches (including approximate or similar) the current feature amount. Search is performed (S305).

  When there is a searched event record (Yes in S306), the related information search unit 125 searches the related database 14 for a related record including the event ID of the event record in the related ID column, and the related ID column of the related record Is acquired (S307). Subsequently, the related information search unit 125 searches the event database for event records related to each event ID included in the acquired related ID string (S308). Subsequently, the search result display unit 126 displays a search result screen on the display device 106 based on the searched event record (S309).

  FIG. 8 is a diagram illustrating a display example of the search result screen. In the same figure, the search result screen 510 displays the file name (still.jpg) of the event record related to the feature quantity that matches the current feature quantity, and the file name, event type, and time of the event record associated with the event record. A list of bands is displayed. In addition, the occurrence date of the event is displayed on the left side of the search result screen 510. In addition, what is necessary is just to change a display item suitably according to operation | use. At least information that allows the operator to identify the electronic file associated with the image data input as the seed of the search (that is, identification information of the electronic file; in this embodiment, the file name) may be displayed. . Alternatively, a thumbnail image of an electronic file may be generated and displayed on the search result screen 510.

  The operator can easily specify an electronic file associated with a handwritten memo or the like by referring to such a search result screen 510.

  Subsequently, when the operator selects a desired line from the list displayed on the search result screen 510, the search result display unit 126 displays the electronic file related to the file name of the selected line on the display device 106. (S310).

  As described above, according to the information management apparatus 10 in the present embodiment, it is possible to appropriately associate a handwritten memo or printed matter with an electronic file that has been handled in the same time zone. Therefore, the user can easily search for an electronic file used when the memo was created based on a handwritten memo or the like.

  The association database 14 is not always necessary. At the time of information retrieval, processing corresponding to registration processing in the association database 14 may be performed. However, by creating the association database 14 when accumulating information, it is possible to quickly search for electronic files and the like related to handwritten memos or printed matter.

  A feature amount extraction method by the feature amount extraction unit 113 or the feature amount extraction unit 123 will be described below. FIG. 9 is a flowchart for explaining a feature amount extraction method according to this embodiment.

  First, an input image (image data from which feature amounts are extracted. Here, it is assumed that the image data is a text document) is binarized (S401). For example, the description described in Japanese Patent No. 4018310 may be used for binarization. Subsequently, distance conversion is applied to the binary image to generate a distance image (S402). The distance conversion is a process of obtaining the shortest distance from each pixel of the binary image to the black pixel and using it as the pixel value. In the distance image, a local maximum appears in the form of a ridge between the gaps and lines of the text document. Subsequently, the maximum value of the distance image is skeletonized to create a connection of lines (ridges) having a width of one pixel. Subsequently, a point where the line intersects (T-shaped intersection, cross intersection, etc.) is defined as a feature point (S404). Therefore, a plurality of feature points can be detected.

  The feature points are in a blank space of the text document or in an area of uniform color. Therefore, the use of the feature points is superior in stability and repeatability. That is, in a text document, the text portion often includes a high frequency image content, and the image component easily deteriorates. On the other hand, the white space (background portion) in the text document is almost stable and includes a component that is almost DC in the frequency domain. Noise in the white space can be easily selected without affecting the characteristic location. The location of the feature points is robust (robust) against noise because text near each feature point usually degrades in the same way.

  Subsequently, the scale (feature scale) of each feature point is determined (S405). The distance conversion value at the feature point is unique to each feature point. Therefore, the feature scale is assigned as a value obtained by multiplying the distance conversion value of the feature point by the factor α.

  The content inside the circle centered on the feature point and radiused on the feature scale is invariant to the in-plane rotation and zoom of the camera (or the distance to the paper document if the camera's optical axis is perpendicular to the paper). is there. This feature is local, i.e. the circle is small, so this feature is also robust against changes in the viewing angle of the camera relative to the paper (out-of-plane rotation). In other words, even if overall projection distortion occurs, such projection distortion is very small locally, and the effect on the feature scale is negligible for wide-range camera out-of-plane rotation. Thus, the feature scale is stable and repeatable, as is the location of the feature points. The feature scale is calculated based on the skeleton and therefore becomes stable.

  Subsequently, for each feature point, the direction (feature direction) in which the skeleton line extends is determined (extracted) (S406). The two-dimensional local coordinate system for each feature point is determined so that the feature point is the origin and the local direction is the x-axis. The feature direction is calculated based on this local coordinate system. In order to derive the feature direction, a circle having a radius determined by the feature scale with the feature point as the center is calculated. The circle intersects the skeleton line at several positions. The feature direction is the position of the two most distant intersection points (among all pairs of points on the circumference that intersect the skeleton). When there are two pairs having similar distances, a pair closer to the horizontal direction is selected. For example, assume that the circle and the skeleton line intersect at three points A, B, and C. When the two points A and B have the longest distance, the direction of the line segment AB is the feature direction.

  Since the local orientation is sensitive to noise and some orientations can even be outliers, global orientation estimation is utilized. In particular, the overall orientation of the image can be estimated from feature points with local directions. This estimation is done by counting the number of local feature directions over 360 bins (once for each bin) (voting). The maximum value of the orientation histogram is selected as the overall direction. After voting, all feature points are reassigned to an orientation that matches the selected overall orientation. In this embodiment, when counting the number of histograms, a coarse division method is used to ensure that the histograms change smoothly even when disturbed by noise.

  The feature direction is determined by the skeleton, and the skeleton is stable as described above. Therefore, the characteristic direction is also stable.

  A set (array) of positions and directions for each feature point derived as described above is a feature amount of the input image.

  Next, a feature value matching method will be described. The feature amount extracted from the two image data to be compared is a set (array) of positions and directions (attribute information of each feature point) for each feature point as described above. Therefore, feature amount matching is performed by matching two arrays. If one array matches at least a part of the other array, it is possible to determine that one image data is at least a part of the other image data and the position of the part. Note that the comparison between the position and direction of the feature point may be determined not only as a complete match but also as a match within a predetermined threshold range.

  According to the above feature amount, not only perfect matching but also similar determination is possible. It is also possible to compare a part with the whole. Specifically, image data of only a part of the paper surface is captured during information storage, and image data of the entire paper surface is captured during information retrieval, or vice versa (image data of the entire paper surface is captured during information storage). Even when a part of the image data of the paper is captured at the time of information retrieval), both can be matched. Therefore, for example, when searching for information, a part of a memo or the like can be easily captured by a mobile phone with a camera or the like, and an electronic file related to the memo or the like can be searched. However, the feature quantity applicable to the present invention is not limited to the following. For example, a hash value or the like may be used as the feature amount.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

It is a figure which shows the hardware structural example of the information management apparatus in embodiment of this invention. It is a figure which shows the software structural example of the information management apparatus in embodiment of this invention. It is a flowchart for demonstrating the process sequence of the information management apparatus at the time of information storage. It is a figure which shows the structural example of an event database. It is a flowchart for demonstrating the search process of the related event record by a related production | generation part. It is a figure which shows the structural example of an association database. It is a flowchart for demonstrating the process sequence of the information management apparatus at the time of an information search. It is a figure which shows the example of a display of a search result screen. It is a flowchart for demonstrating the extraction method of the feature-value in this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Information management apparatus 11 Information storage part 12 Information search part 13 Event database 14 Association database 15 OS (Operating System)
100 drive device 101 recording medium 102 auxiliary storage device 103 memory device 104 CPU
106 display device 107 input device 108 projection device 109 image input device 111 event acquisition unit 112 rasterization unit 113 feature quantity extraction unit 114 image acquisition unit 115 image composition unit 116 information recording unit 117 relation generation unit B bus

Claims (12)

An information management device having information storage means and information search means,
The information storage means includes
Operation time recording means for associating the identification information of the electronic data with the time information indicating the time of occurrence of the operation in association with the time of occurrence of the operation on the electronic data;
First image data acquisition means for acquiring, from an image input device, image data obtained by capturing at least a part of a handwritten paper document;
First feature amount extraction means for extracting feature amounts of the image data;
Feature amount recording means for associating and recording the time information indicating the acquisition time of the image data and the feature amount in the storage device;
The information search means includes
Second feature amount extraction means for extracting feature amounts of image data input from the image input device;
Search means for searching the storage device for feature quantities that match the extracted feature quantities;
Related information acquisition means for acquiring, from the storage device, the identification information associated with the time information relating to the time within a predetermined period from the time indicated by the time information associated with the searched feature amount;
An information management apparatus comprising display control means for displaying information based on the acquired identification information. The operation time recording means records the electronic data identification information and the time information indicating the output time in the storage device in association with each other when the image data related to the electronic data is output from the output device,
The first feature amount extraction unit extracts a feature amount of the image data output from the output device,
The information management apparatus according to claim 1, wherein the feature quantity recording unit records the timing information indicating the output timing of the image data in association with the feature quantity in the storage device. The information storage means includes
The information processing apparatus includes an association unit that searches for a matching feature quantity among the feature quantities recorded in the storage device and records association information of information related to the searched feature quantity in the storage device. The information management apparatus according to 1 or 2.   The information management apparatus according to claim 1, wherein the feature amount is a set of attribute information related to a plurality of feature points in the image data. An information management method by an information management apparatus having an information storage means and an information search means,
An operation timing recording procedure in which the information storage means records in the storage device the identification information of the electronic data and the timing information indicating the occurrence timing of the operation in association with the occurrence of the operation on the electronic data;
A first image data acquisition procedure for acquiring, from the image input device, image data in which the information storage means captures at least a part of a handwritten paper document;
A first feature amount extraction procedure in which the information storage means extracts a feature amount of the image data;
A feature amount recording procedure in which the information storage means records the time information indicating the acquisition time of the image data in association with the feature amount in the storage device;
A second feature amount extraction procedure in which the information search means extracts a feature amount of image data input from an image input device;
A search procedure in which the information search means searches the storage device for a feature quantity that matches the extracted feature quantity;
A relationship in which the information search means acquires from the storage device the identification information associated with the time information relating to a time within a predetermined period from the time indicated by the time information associated with the searched feature amount. Information acquisition procedure;
An information management method comprising: a display control procedure in which the information search means displays information based on the acquired identification information. In the operation time recording procedure, when the image data related to the electronic data is output from the output device, the electronic data identification information and the time information indicating the output time are recorded in the storage device in association with each other,
The first feature amount extraction procedure extracts a feature amount of image data output from the output device,
6. The information management method according to claim 5, wherein in the feature quantity recording procedure, timing information indicating a timing of outputting the image data and the feature quantity are associated and recorded in the storage device.   The information storage means has an association procedure for searching for a matching feature amount among the feature amounts recorded in the storage device and recording association information of information relating to the searched feature amount in the storage device. 7. The information management method according to claim 5 or 6, wherein:   The information management method according to claim 5, wherein the feature amount is a set of attribute information regarding a plurality of feature points in the image data. A program for causing a computer to function as information storage means and information search means,
The information storage means includes
Operation time recording means for associating the identification information of the electronic data with the time information indicating the time of occurrence of the operation in association with the time of occurrence of the operation on the electronic data;
First image data acquisition means for acquiring, from an image input device, image data obtained by capturing at least a part of a handwritten paper document;
First feature amount extraction means for extracting feature amounts of the image data;
Feature amount recording means for associating and recording the time information indicating the acquisition time of the image data and the feature amount in the storage device;
The information search means includes
Second feature amount extraction means for extracting feature amounts of image data input from the image input device;
Search means for searching the storage device for feature quantities that match the extracted feature quantities;
Related information acquisition means for acquiring, from the storage device, the identification information associated with the time information relating to the time within a predetermined period from the time indicated by the time information associated with the searched feature amount;
A display control means for displaying information based on the acquired identification information. The operation time recording means records the electronic data identification information and the time information indicating the output time in the storage device in association with each other when the image data related to the electronic data is output from the output device,
The first feature amount extraction unit extracts a feature amount of the image data output from the output device,
The program according to claim 9, wherein the feature amount recording unit records the timing information indicating the output timing of the image data in association with the feature amount in the storage device. The information storage means includes
The information processing apparatus includes an association unit that searches for a matching feature quantity among the feature quantities recorded in the storage device and records association information of information related to the searched feature quantity in the storage device. The program according to 9 or 10.   The program according to claim 9, wherein the feature amount is a set of attribute information related to a plurality of feature points in the image data.

Images