JP6151989B2 - Information presenting apparatus, information presenting system, information presenting method, and program - Google Patents

Description

  The present invention relates to an information presentation apparatus, an information presentation system, an information presentation method, and a program.

  For example, when an important contract such as a loan contract is performed in a bank, a process is used in which a contractor provides a legal explanation to a customer face-to-face, the customer agrees to it, and creates a document on that. At this time, the customer often fills in the document by hand, but a customer who is not accustomed to document creation may fail to complete the document.

  In particular, with respect to documents that are entered by both the merchant and the customer, in order to save time, a document that the merchant filled in first may be prepared in advance. In this case, the document is completed only by the customer filling in on the contract conclusion date. However, if the customer fails to fill in the document, the document needs to be recreated, which takes a lot of time.

  As a device for preventing a document entry mistake, Patent Document 1 proposes the following. That is, the entry frame obtained by analyzing the document image to be filled in is compared with the position of the writing instrument on which the user performs handwriting, and it is determined whether the entry position is correct as the entry target.

JP 2012-151753 A

  However, although the technique described in Patent Document 1 is effective in preventing errors in the entry position, it is difficult to prevent errors in the entry contents themselves.

  An object of the present invention is to reduce entry errors when filling in a document.

In order to achieve the object of the present invention, for example, an information presentation apparatus of the present invention comprises the following arrangement. That is,
Image obtaining means for obtaining a document image obtained by photographing a first document including an entry area that can be filled in by the user so that the user also appears;
A specifying means for specifying the position of the entry area;
Information acquisition means for acquiring reference information corresponding to the entry area;
Analyzing the document image to determine an arm used by the user for writing;
Control means for controlling the presenting means to present the reference information to the user so as to be viewed in the vicinity of the entry area according to the determination result by the arm determining means;
Equipped with a,
The first document includes a plurality of entry areas;
The identifying means identifies an entry area that the user intends to fill out of the plurality of entry areas,
The information acquisition means acquires reference information corresponding to the specified entry area,
The control means controls the presenting means to present the reference information to the user so as to be visually recognized in the vicinity of the specified entry area,
The specifying means specifies the entry area as the entry area that the user intends to fill in when the distance between the tip of the entry tool used by the user and the entry area is equal to or less than a threshold value. Features.

  It is possible to reduce mistakes when filling out documents.

The figure which shows the structure of the information presentation apparatus which concerns on one Embodiment. The figure explaining the effective usage example of the information presentation apparatus which concerns on one Embodiment. The figure which shows the structural example of the data used in one Embodiment. The figure which shows the structural example of the data used in one Embodiment. The flowchart which shows an example of the information presentation method which concerns on one Embodiment. The flowchart of each process performed with the information presentation method which concerns on one Embodiment. The flowchart of the layout determination process which concerns on one Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. However, the scope of the present invention is not limited to the following examples.

[Embodiment 1]
FIG. 1A is a block diagram illustrating an overall configuration of the information presentation apparatus 109 according to the first embodiment. The information presentation device 109 includes a calculation unit 101, a projection unit 102, a photographing unit 103, a UI unit 104, a communication unit 105, a bus (BUS) 106, a random access memory (RAM) 107, and a storage 108.

  The calculation unit 101 controls the overall operation of the information presentation device 109. The calculation unit 101 can be a microprocessor, for example. The calculation unit 101 performs calculation or logical determination for image or video shooting processing, information projection processing, image identification processing, image correction processing, image editing processing, or character recognition processing.

  The projection unit 102 presents information to the user by projecting various types of information such as characters, images, and videos. In the present embodiment, the projection unit 102 is a projector and superimposes information on a paper document. However, the projection unit 102 is not limited to a projector, and may be, for example, a head-mounted display having an AR (Augmented Reality) function. Such a head mounted display can also project information to the user's eyes. Thus, in the present embodiment, when information is superimposed and displayed on a paper document in the real space, in addition to a method of projecting information on the paper document in the real space, a paper document image in the real space and A method of visually superimposing information and presenting it to the user can also be used. In addition, the information display method according to the present embodiment is not limited to a method of superimposing and displaying information on a paper document in the real space, and information may be superimposed and displayed on a paper document in the virtual reality space. Hereinafter, the process of superimposing and displaying information on a paper document is referred to as a “projection” process for convenience.

  The photographing unit 103 reads an image including a still image or a moving image. For example, the imaging unit 103 can be an imaging sensor that reads an image in the real space. The photographing unit 103 can convert the read image into electronic data.

  The UI unit 104 receives an instruction from the operator. The UI unit 104 can be, for example, a button or a touch panel. However, the UI unit 104 does not have to be a physical button or a touch panel. For example, the operator may input an instruction when the projection unit 102 projects an image such as a virtual button or a touch panel, and the operator touches the image. In this case, the UI unit 104 may receive the result obtained by the calculation unit 101 analyzing the image data obtained by the photographing unit 103 as an instruction from the operator. In this way, a virtual UI can be realized.

  The communication unit 105 is, for example, a network controller, and exchanges data with the outside via a wired or wireless communication path.

  The BUS 106 transfers signals or data between the units included in the information presentation device 109. The signal includes, for example, an address signal or a control signal for the arithmetic unit 101 to control each unit.

  The RAM 107 is a readable / writable random access memory. The RAM 107 is used as a primary storage for various data received from each unit. Specifically, the RAM 107 has various work areas for storing data necessary for processing. The RAM 107 stores various data whose values are changed during processing.

  The storage 108 is a nonvolatile memory. The storage 108 can be, for example, a removable medium such as an SD card or a CF card, or a large-capacity memory built in the information presentation device 109. The storage 108 may be a hard disk, for example. The storage 108 can store a large amount of data in a changeable manner. A control program executed by the arithmetic unit 101 is stored in the storage 108. The storage 108 can store a boot program, various initial data, an image identification program, an image correction program, an image editing program, a character recognition program, and the like. The calculation unit 101 can load a program stored in the storage 108 into the RAM 107 and execute it as necessary. According to such a configuration, it is not necessary to fix the program, and the program can be configured to be modifiable, so that a flexible system can be realized.

  In the present embodiment, the information presentation device 109 includes a projection unit 102 and a photographing unit 103. However, the information presentation device 109 and the projection device including the projection unit 102 may be different devices. In addition, the information presentation device 109 and the imaging device including the imaging unit 103 may be different devices. Such an information presentation device, a projection device, and a photographing device constitute an information presentation system.

  FIG. 1B is a block diagram illustrating a functional configuration of the information presentation device 109 according to the first embodiment. The information presentation device 109 includes an image acquisition unit 151, a document recognition unit 152, an area specification unit 153, an information acquisition unit 154, a layout determination unit 155, and a presentation control unit 156. The functions of these units can be realized by the operation unit 101 operating according to a program as described above. However, the information presentation device 109 may have a dedicated circuit that operates as at least one of these units. The operations of these units will be described later with reference to FIGS.

  Next, an effective use example of the information presentation apparatus according to the present embodiment will be described with reference to FIG. FIG. 2 shows an example in which the information presenting apparatus according to the present embodiment is used in contract work at a bank. In FIG. 2, the user is about to create a document by handwriting.

  The information presentation device 109 according to the present embodiment includes the projection unit 102 and the imaging unit 103 as described above. User 202 who is an explainer is a bank employee and is in charge of contract work. The user 202 sits facing the user 203 who creates a document with the desk 210 in between. The user 202 leads the document creation in the contract work described below. A user 203 is a bank customer and uses a writing tool such as a ballpoint pen 206 to create a first document 205, which is a contract document, by hand as will be described below. Hereinafter, the document 205 in which characters are entered may be referred to as an entry target document.

  The document 205 includes an entry target area 209 that is an entry area that the user 203 can enter by handwriting. Representative examples of the entry target area 209 include a customer's address and name. These items must be filled in accurately. With regard to customer addresses in particular, due to legal reasons, the address in the current address column described in official documents such as resident's cards may have to match the written address one by one. . In addition, regarding the address, there are cases where the address notation is omitted or not. Furthermore, when a long sentence such as an address including an apartment name is written, the number of characters to be written may be several tens of characters. In this case, it is not easy to write characters by handwriting in a well-balanced manner in the entry target area 209 of different sizes depending on documents. Especially for the user 203 who does not usually create documents, such an entry work becomes a burden.

  According to the present embodiment, in such a situation, it is possible to reduce the possibility of writing errors when the user 203 creates a handwritten document.

  The second document 204 is an official document such as a resident's card and includes an address field 207. This address field 207 describes a character string of an address to be entered in the entry target area 209. Hereinafter, the document 204 used as a model for entry may be referred to as a model document. When the document 204 is placed on the desk 210, document recognition processing using the photographing unit 103 built in the information presentation device 109 is executed. Specifically, an analysis using character recognition processing is performed on a document image including the document 204 acquired by the photographing unit 103. As a result, document information such as the document name, name, and address described in the document 204 is extracted as digitally processable data. The document information extracted in this way is used for projection by the projection unit 102. The extracted document information is stored in the storage 108 or a server connected via the network. The stored document information can be acquired at an arbitrary timing.

  Here, the user 203 picks up the ballpoint pen 206 and tries to enter a handwritten address in the entry target area 209 in the document 205. Then, the information presentation device 109 performs document recognition processing on the document image including the document 205 photographed by the built-in photographing unit 103. This document image may include both the document 204 and the document 205.

  In addition, the information presentation device 109 performs a region specifying process using a document image including the document 205 photographed by the built-in photographing unit 103. In the area specifying process, the user's action is detected by performing an image identification process, and the area to be filled in is specified by the user 203 to enter characters. In the present embodiment, an area in which the distance between the tips of the ballpoint pens is equal to or less than a threshold value is determined as the entry target area. Specifically, by analyzing the spatial coordinates, when it is determined that the tip of the ballpoint pen 206 is in contact with the entry target area 209, it is determined that the user 203 has an intention to fill in the address field. The As will be described later, the position of each region on the document 205 is determined by the document recognition unit 152.

  The method for specifying the entry target area is not limited to this method. For example, the entry target area may be specified according to the distance on the image between the tip of the ballpoint pen 206 and each area on the document 204. Specifically, when the distance between the tip of the ballpoint pen 206 and a certain area on the document 204 is within a threshold value, this area can be specified as the entry target area. Further, instead of or in addition to the image photographed by the photographing unit 103, information from a sensor that detects the position and orientation of the user 203, the ballpoint pen 206, or the like may be used to specify the entry target area. .

  Moreover, you may specify the area | region which the user touched directly with the finger | toe as an entry object area | region. The position of the user's finger can be determined, for example, by performing image recognition processing on the image captured by the imaging unit 103. As a further method, the UI unit 104 may detect a user selection for the entry target area using a virtual button or the like. In this case, the entry target area is specified according to the detected user selection. For example, a list of field names of each area projected by the projection unit 102 can be used as a virtual button, and the user can make a selection by directly touching this list with a finger.

  When the entry target area is specified by the area specifying process, the information acquisition unit 154 acquires the character string to be entered by the user 203. In the present embodiment, highly relevant information is selected from the document information extracted by the document recognition process. Specifically, information that matches the area name of the entry target area 209 is selected. Here, the entry target area 209 is an address field, and document information indicating an address is selected. Then, the character string included in the selected information is specified as a model character string. In this manner, the information acquisition unit 154 acquires a character string that is predicted to be entered by the user 203.

  When the model character string is specified, the layout determination unit 155 presents the character string such as the character size or the line feed position so that the model character string fits in the entry area according to the size of the entry target area 209. Determine the layout. The model character string for which the layout is determined is projected as reference information 208 in the vicinity of the entry target area 209 using the projection unit 102 by the presentation control process.

  According to such a configuration, for example, when it is necessary to describe an address as described in an official document such as a resident's card in a contract, particularly when the address includes a portion that is easily omitted such as an address, It is possible to reduce entry errors. In addition, by projecting a character string whose layout is appropriately adjusted, it becomes easy to enter a long address including an apartment name in a predetermined frame with an appropriate character size.

  The projection position of the reference information 208 may be within the entry frame of the entry target area 209 or outside the entry frame. In this case, the projection location can be determined or changed according to the presence / absence of a blank area included in the document 205 and the dominant hand determination result of the user 203 using human body recognition processing. The dominant arm determination can be performed by, for example, an arm determination unit (not shown) analyzing an image including the document 205 and the arm of the user 203 by image recognition processing. In accordance with this result, the presentation control unit 156 described later controls the projection position of the reference information 208.

  Specifically, when the user's 203 dominant hand is recognized as the right arm, the reference information 208 can be projected on the left side with respect to the visual front of the user 203 so as not to interfere with the writing operation by the user 203. . Further, when it is determined that the dominant arm of the user 203 is the left arm, the reference information 208 can be projected on the right side. When the reference information 208 is projected into the entry frame, a projection unit that projects information from the back of the paper, such as a display, may be installed on the desk, and information may be projected using this projection unit. Further, when there is not a sufficient blank area on the document 205, highlighting may be performed by a method such as reversing the background of the character string to be projected and the character color to improve visibility. Further, the projection position of the reference information 208 may be outside the document 205, for example, on the desk 210.

  In this description, it is assumed that the reference information 208 is a character string. However, reference information including images or videos can also be projected. For example, a seal type such as a real seal or a bank seal may be displayed at a place where a seal is required. With such a configuration, the possibility of forgetting the seal can be reduced. In this description, the projection unit 102 such as a projector projects the reference information 208. However, as described above, the same effect can be obtained by displaying the reference information 208 on a head mounted display or the like having an AR (Augmented Reality) function. In another embodiment, corresponding reference information may be projected on a plurality of entry areas, for example, on all entry areas.

  According to the above method, the character string itself to be entered by the user 203 is displayed at hand in a state of fitting to the entry target area 209 at least one of before entry and during entry. For this reason, it is possible to reduce the occurrence of writing errors due to erroneous writing or poor character size balance.

  FIG. 3 shows a data configuration example of various processing information used in processing performed by the information presentation apparatus 109 according to the present embodiment. FIG. 3A shows a configuration example of document information extracted by the document recognition unit 152 as described above. Specifically, the document information is composed of a plurality of data composed of combinations of information types such as document names or issue dates, text data of information, and the number of text characters. Here, the name data 301 indicates the name of the user 203 described in the document 204. The address data 302 indicates the address of the user 203 described in the document 204.

  FIG. 3B shows a configuration example of region setting information of the document 205 used by the information acquisition unit 154 to acquire information. This area setting information is prepared in advance according to the type of the document 205 that is supposed to be filled in, and is stored in the storage 108. Each data corresponds to each entry area on the document 205, and includes a field ID, a document name, a field name, field coordinates, corresponding data, a projection action, and the like. The field ID is an ID for each entry area.

  Data 303 and data 304 are data relating to different entry areas “current address” and “contractor name” in the document 205 (for example, mortgage contract A) having the same document name. Data 305 and data 306 are data relating to different entry areas “current address” and “contractor name” in the document 205 (for example, car loan contract B) having different document names. In these data, “field coordinates” that specify the range of an entry area (for example, entry field) on the document 205 are set.

  The area specifying unit 153 refers to the area setting information, selects an entry area whose distance from the tip of the ballpoint pen 206 is equal to or less than a threshold, and specifies this entry area as an entry target area. In the present embodiment, when the coordinates of the tip position of the ballpoint pen 206 are in contact with the document 205 within the range specified by the field coordinates, the area specifying unit 153 is a field name entry area corresponding to the field coordinates. Is specified as the target area. In one embodiment, “field coordinates” are expressed in a coordinate system with reference to the document 205. In this case, for example, by detecting the position of the document 205 from the image photographed by the photographing unit 103, the coordinates on the image can be converted into coordinates based on the document 205.

  As another method, the area specifying unit 153 recognizes an entry frame and its label near the tip position of the ballpoint pen 206 by analyzing the image of the document 205, and treats it as an entry target area and its field name, respectively. Also good.

  When the area specifying unit 153 specifies an area to be filled in intended by the user 203 in the document 205, the information acquisition unit 154 refers to the area setting information and acquires “corresponding data”. Specifically, “corresponding data” corresponding to the field name of the entry target area and the document name of the document 205 is acquired. “Correspondence data” specifies the document name of the document 204 and the type of information to be projected. For example, when the area specifying unit 153 specifies the data 303, that is, the entry area “current address” of the “mortgage contract A”, the information acquisition unit 154 stores the corresponding data “$ resident card. $ Address”. Get the value. That is, the information acquisition unit 154 acquires the data of the type “address” for the document name “resident card”, that is, the data 302 from the document information as a model character string that the user 203 intends to enter.

  The data acquired by the information acquisition unit 154 is projected on the document 205 by the projection unit 102 according to the projection action corresponding to the entry target area. For example, when the entry target area is the entry area “current address” of “mortgage contract A”, projection is performed according to the projection action of pattern A. The projection action specifies a method of projecting information by the projection unit 102. The type of projection action corresponding to the entry area is defined in the area setting information. A specific projection method corresponding to each projection action is defined by projection action definition information. FIG. 3C shows a configuration example of the projection action definition information. A plurality of patterns may be defined for a single entry area.

  Data 307 is definition data regarding the projection action of pattern A. In pattern A, the “projection position” of information is at the top of the field. That is, information is projected on the top of the entry target area. In the pattern A, “action definition” is “display character replacement”. Specifically, among the model character strings acquired by the information acquisition unit 154, characters already entered by the user 203 are replaced with a symbol “◯”, and the replaced character string is projected. Characters already entered by the user 203 can be recognized by a character recognition unit (not shown) performing character recognition processing on the image of the document 205 captured by the imaging unit 103. As described above, the projecting unit 102 can project the reference information so that characters already entered in the model character string can be distinguished from characters that have not yet been entered. In this case, since the character string to be projected changes according to the entry status of the user 203, the user 203 can easily determine what the character to be entered next is.

  Data 308 to 311 are other projection action definition information. In the pattern B defined by the data 308, the “projection position” of the information is the upper part of the field, and the “action definition” is “display color change”. Specifically, in the model character string acquired by the information acquisition unit 154, characters already entered by the user 203 are grayed out (grayed out), and unfilled characters are blue. In this case, depending on the state of entry by the user 203, an unfilled character string projected on the filling target area is emphasized with respect to the filled one, and thus the user 203 has been filled in. It becomes easy to distinguish character strings.

  In the pattern C defined by the data 309, the “projection position” of information is in the field, and the “action definition” is “correctness check”. Specifically, whether or not the character string that the user 203 has entered in the target area matches the character string specified by “corresponding data”, for example, the character string corresponding to “$ resident card. $ Address”. Is checked. In this way, it is possible to check at the time of entry whether or not the entered character string includes an error. If an error is included, the projection unit 102 can notify the user 202 or the user 203 by a method such as projecting a warning display near the entry target area. Such notification is also included in the reference information presented by the information presentation device 109.

  In the pattern D defined by the data 310, “projection position” is at the top of the field, and “action definition” is “correction”. In “correction”, the character string already entered by the user in the entry target area and the size of the character string are recognized. Further, the size of the projected character string corresponding to the recognized character string is determined. The corresponding projected character string is a character string in the model character string corresponding to the recognized character string. Then, the size of the recognized character string is compared with the size of the projected character string, and the result of the comparison is notified to the user. Such a comparison result is also included in the reference information presented by the information presentation device 109.

  In a specific example, at least a part of the character region area of the entered character string is recognized and compared with the character region area of the projected character string corresponding thereto. The area of the character region of the character string may be, for example, a rectangular area surrounding the character string, but the calculation method is not particularly limited. Then, according to the degree of coincidence between the character area of the entered character string and the character area of the corresponding projected character string, for example, the difference or ratio, the projected character string or the background of the projected character string The user is notified by changing the color.

  For example, when the degree of coincidence between the character region area of the entered character string and the character region area of the corresponding projected character string is high, for example, when the error is within a threshold, the projected character string or projection The background color of the character string to be set is blue. In addition, when the degree of coincidence for a newly entered character is lower than the degree of coincidence for an already entered character, the projected character string or the background color of the projected character string is yellow. . Further, when characters having low matching degrees are continuously entered, the projected character string or the background color of the projected character string is red. The user 203 can adjust the size of the character to be entered next with reference to information according to the character entry status.

  In the pattern E defined by the data 311, “projection position” is in the field, and “action definition” is an alert. Specifically, for example, when the entry target area is the amount column, the image is projected so that the amount column is red. In this way, the user 203 can be alerted so as not to enter an incorrect amount. Such alerting information is also included in the reference information presented by the information presentation device 109.

  FIG. 4 shows a data configuration example of processing information used in the layout determination processing in the present embodiment. The field ID 401 is the same as that shown in FIG. FIG. 3B shows a field ID 001 for the data 303. The model character string 402 and the number of characters 403 are the same as the data 302 in FIG. 3A corresponding to the data 303 in FIG. This model character string 402 is a character string acquired by the information acquisition unit 154.

  The layout determining unit 155 determines a layout such as a character size or a line feed position so that the model character string 402 is projected in a size that fits in the entry target area, that is, the field coordinates indicated in the data 303. FIG. 4 shows the recommended character size 404 and the recommended line feed position 405 determined by the layout determining unit 155. The recommended character size 404 and the recommended line feed position 405 may be determined so that a line break is not performed in the middle of an apartment name or a place name by performing word analysis processing or the like when determining the layout.

  In addition, when there is no appropriate projection area on the document 205 because the document 205 includes a lot of character strings or figures, the number of characters or the character size can be identified instead of projecting the character string. Symbols such as “◯” or “*” may be presented as reference information. In this case, even when the character string is projected on the document and it is difficult to interpret the character, the recommended balance of the character string can be grasped.

  The model character string 402 is projected onto the current address entry area of the document 205 by the projection unit 102 in accordance with the layout determined by the layout determination unit 155 and the defined projection action in the presentation control process.

  FIG. 5 is a flowchart illustrating a processing procedure of the calculation unit 101. The overall operation of the information presentation device 109 is controlled by the processing shown in FIG. Step S501 is a system initialization process, and the calculation unit 101 initializes various parameters. Step S502 is processing in which the calculation unit 101 waits for the UI unit 104 or the imaging unit 103 to detect some event. Specifically, the calculation unit 101 waits for detection of an event such as a user operating a physical button or a user touching a virtual button or a projection area. Determination of a user's contact with a virtual button or projection area can be realized using the projection unit 102 and the imaging unit 103 as described above.

  When an event occurs, in step S503, the calculation unit 101 determines the event that has occurred. And the calculating part 101 starts the various processes according to the kind of event. Processes corresponding to various events are collectively expressed as step S504. The processing in step S504 includes document recognition processing, area identification processing, information acquisition processing, layout determination processing, presentation control processing, and the like. Step S504 also includes a termination process in response to a power OFF event. In the following description, it is assumed that these processes are performed by each unit realized by the arithmetic unit 101 operating according to a program.

  FIG. 6A is a flowchart of document recognition processing for the document 204, which is an example of processing executed in step S504. In this processing, the document recognition unit 152 acquires an image of the document 204 that is a model document, and specifies the document 204. In the present embodiment, as described above, candidates for information to be projected are extracted from the document 204 that is a model document.

  In step S601, the image acquisition unit 151 acquires an image of the document 204 as data that can be digitally analyzed. The document image is captured by the imaging unit 103 when a new document enters the imaging range of the imaging unit 103. The image acquisition unit 151 transfers the acquired image of the document 204 to the document recognition unit 152.

  In step S <b> 602, the document recognition unit 152 performs area recognition and character recognition on the image of the document 204. Through this process, the document recognition unit 152 extracts document information such as a document name, an address, and a date from each area included in the document 204. In this way, the document recognition unit 152 acquires a candidate for information to be projected. In one embodiment, the document recognition unit 152 can extract, for each region detected from the document 204 by the region recognition process, a label attached to the region and a character string described in the region.

  In another embodiment, the document recognition unit 152 may extract document information from the document 204 with reference to area information about a model document prepared in advance. The region information specifies, for example, the position of the region in the document and the field name for each region for each type of model document. The document recognition unit 152 can determine the type of the document 204 (for example, the document name) and the position of the document 204 in the image using a technique such as pattern matching. Then, the document recognition unit 152 refers to the area information corresponding to the determined type of the document 204, and extracts information such as a character string included in each area. In step S603, the information extracted in this way is recorded in the storage 108 in association with the field name and the document name as shown in FIG. But the information to extract is not limited to a character string, For example, an image may be sufficient.

  In a further embodiment, the region from which document information is extracted may be specified by user input. For example, the document recognizing unit 152 can perform character recognition processing on an area within a range designated by user input, and obtain a character string as a candidate for information to be projected.

  In step S603, the document recognition unit 152 stores the character recognition result in step S602. Specifically, the document recognition unit 152 stores the document information extracted for each document area in the document 204 in a database in the storage 108. Thereafter, the process returns to the flow of FIG.

  In the present embodiment, the document recognition unit 152 extracts document information, that is, candidates for information to be projected, from the document 204 arranged on the desk 210. However, candidates for information to be projected may be prepared in advance and recorded in a database in the storage 108.

  Here, the case where one copy of the document 204 as a model document exists on the desk 210 has been described as an example. When a plurality of documents such as old and new resident cards exist on the desk 210, the user may be inquired about which document information is to be extracted. For example, the document recognition unit 152 may use the projection unit 102 to project a message asking which document information is to be extracted from. The user 203 can select a document, for example, by touching the document directly or by selecting a document name from the displayed list of document names. These user selections can be detected by the UI unit 104 as described above.

  FIG. 6B is a flowchart of document recognition processing, region identification processing, information acquisition processing, layout determination processing, and presentation control processing, which is an example of processing executed in step S504.

  Step S604 is a document recognition process for the document 205, which is an entry target document. In step S <b> 604, the image acquisition unit 151 acquires an image of the document 205. The image acquisition unit 151 transfers the acquired image of the document 205 to the document recognition unit 152. The document recognition unit 152 analyzes the transferred image and detects that the document 205 is placed on the desk 210. Thereafter, the document recognition unit 152 determines the type (for example, document name) of the document 205 by performing region recognition and character recognition on the image.

  For example, the type of the document 205 can be determined by recognizing the title of the document 205. However, the method for determining the type of the document 205 is not limited to this method. For example, the user may be inquired about the type of the document 205. As a specific example, the UI unit 104 can detect a user selection for the type of the document 205 using a virtual button or the like, and can transmit the detected user selection to the document recognition unit 152. In this case, the list of document names of the entry target documents projected by the projection unit 102 can be used as a virtual button, and the user makes a selection by directly touching the list with a finger. Can do. If the document recognition processing for the document 205 has already been completed, the processing in step S604 may be omitted.

  In another embodiment, the document recognition unit 152 may determine the type of the document 205 and the position of the document 204 in the image using a technique such as pattern matching, as in the document recognition process for the document 204. Further, the document recognition unit 152 may detect an entry area such as an entry frame from the document 205 by area recognition processing, and determine the position on the image or the position in the three-dimensional space. In this case, the document recognition unit 152 may extract, as the field name, the label attached to the entry area for each detected entry area.

  Step S605 is an area specifying process. In step S605, the area specifying unit 153 specifies the position of the entry area included in the document 205 with reference to the area definition information shown in FIG. Further, as described above, the area specifying unit 153 detects the user's behavior and specifies an entry target area to be filled in by the user from among a plurality of entry areas. In addition, the area specifying unit 153 acquires a projection action set in advance for the specified entry target area with reference to the projection action definition information shown in FIG.

  Steps S606 and S607 are information acquisition processing, and the information acquisition unit 154 acquires a model character string intended by the user as reference information as described above. Specifically, in step S606, the information acquisition unit 154 determines whether or not the document information generated in step S603 corresponding to the specified entry target area exists in the database included in the storage 108. For example, the information acquisition unit 154 refers to, for example, the document information in FIG. 3A corresponding to the document name of the document 205, and determines whether there is data having the field name of the entry target area as “type”. Determine. If the corresponding information exists in the database, the process proceeds to step S607, and the information acquisition unit 154 acquires the corresponding character string from the database in the storage 108. If there is no corresponding information, the process in FIG. 6B ends, and the process returns to the flow in FIG.

  In the present embodiment, the information acquisition unit 154 refers to document information stored in a database included in the storage 108 and determines a model character string. However, the method for determining the model character string is not limited to this method. For example, a list of document information obtained by performing processing such as character recognition on the image of the document 204 placed on the desk 210 is projected onto the document 205 or the like, and the model character string is projected to the user. May be selected. Alternatively, the user may select a specific area on the document placed on the desk 210 by touching it, and the character recognition result of the character string included in the specific area may be specified as a model character string.

  Furthermore, the information acquisition unit 154 may project the document information, address information, and the like stored in advance in the database as a list onto the document 205 using the projection unit 102. The information acquisition unit 154 can acquire a user selection for a candidate character string via the UI unit 104, and may use a character string according to the user selection as a model character string.

  Further, the model character string determined by the above-described method may be projected as it is later, or may be edited according to the user's request. When editing, the user receives an editing instruction from the user via the UI unit 104 by, for example, pressing the edit button, and edits the character string to the user using a physical keyboard or a virtually displayed keyboard. You may let them.

  If the document information cannot be acquired in step S606, the process may not be terminated, but a default number of characters or symbols preset according to the entry target area may be projected. For example, when the entry target area is an address field, the number of characters to be entered is considered to be about 30 characters at the maximum. In such a case, by displaying 30 symbols such as “◯” in a layout that enters the entry target area, it becomes possible to enter characters with a character balance suitable for the user. As described above, the information acquisition unit 154 may acquire, as reference information, information including a predetermined number of symbols for the entry target area, and the reference information has an appropriate layout according to the entry target area. It can be projected by the projection unit 102.

  Step S608 is a layout determination process. In step S608, the layout determining unit 155 determines the layout of the character string so that the example character string determined in step S607 fits in the entry target area specified in step S605. The specific method is as already described.

  Step S609 is presentation control processing. In step S609, the presentation control unit 156 controls the projection unit 102 to project the model character string acquired in step S607 onto the document 205 as reference information. At this time, the model character string is presented so as to be visually recognized by the user in the vicinity of the entry target area specified in step S605 according to the projection action acquired in step S605 with the layout determined in step S608. . Thereafter, the process returns to the flow shown in FIG.

  In the present embodiment, it is assumed that the projection range by the projection unit 102 is known. That is, the presentation control unit 156 projects the desired information to a desired position on the document 205 or a position in the real space corresponding to the desired position on the image photographed by the photographing unit 103. 102 can be controlled. When the projection unit 102 is a head mounted display or the like, the image including the document 205 or the real space image and the reference information are presented to the user in a superimposed state. In this case, the projection unit 102 can display an image so that the reference information is presented at a position according to the projection action according to the detected position of the document 205.

  FIG. 7 shows in detail an example of processing performed when projecting a model character string in accordance with the projection action in step S609. FIG. 7 illustrates a case where the projection action of “pattern A” shown in the data 307 of FIG. 3C is used.

  Step S701 is a completed character recognition process. In step S <b> 701, the presentation control unit 156 recognizes characters already entered by the user by analyzing the image acquired by the imaging unit 103. Step S702 is a character display change process. In step S702, the presentation control unit 156 sequentially changes the display of the characters corresponding to the completed characters recognized in step S701 among the model character strings determined in step S607. Specifically, in accordance with the projection action of “Pattern A”, the character corresponding to the entered character in the model character string is replaced with the symbol “◯”. Thereafter, in step S703, the presentation control unit 156 projects the model character string after the processing in step S702 onto the document 205. As described above, the projection position is determined according to the projection action definition information.

  Step S704 is a character number comparison process. In step S704, the presentation control unit 156 compares the number of completed characters recognized in step S701 with the number of characters in the model character string determined in step S607, so that all the model character strings are entered by the user. Determine whether it was done. If the entry has been completed, the process proceeds to step S705. If the entry has not been completed, the process returns to step S701. At this time, the presentation control unit 156 may determine whether or not an unrecognizable character exists in the character string entered by the user, and may notify the user if there is an unrecognizable character. For example, an alert message that identifies unrecognizable characters can be projected so that the user can see it. According to this configuration, it is possible to reduce a possibility that characters are written in an unreadable font and the document has no legal effect.

  Step S705 is reference information update processing. When the entry to the entry target area is completed, the presentation control unit 156 finishes projecting the model character string corresponding to the entry target area. According to the above processing, since the display of the character string changes according to the user entry status, it is easy to determine what character the user should enter next.

  The projection actions of “Pattern B” to “Pattern E” shown in FIG. 3C can also be realized by the same processing as the flowchart shown in FIG. For example, in step S702, the process according to the description of the projection actions of “pattern B” to “pattern E” described above may be performed.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (13)

Image obtaining means for obtaining a document image obtained by photographing a first document including an entry area that can be filled in by the user so that the user also appears;
A specifying means for specifying the position of the entry area;
Information acquisition means for acquiring reference information corresponding to the entry area;
Analyzing the document image to determine an arm used by the user for writing;
Control means for controlling the presenting means to present the reference information to the user so as to be viewed in the vicinity of the entry area according to the determination result by the arm determining means;
Equipped with a,
The first document includes a plurality of entry areas;
The identifying means identifies an entry area that the user intends to fill out of the plurality of entry areas,
The information acquisition means acquires reference information corresponding to the specified entry area,
The control means controls the presenting means to present the reference information to the user so as to be visually recognized in the vicinity of the specified entry area,
The specifying means specifies the entry area as the entry area that the user intends to fill in when the distance between the tip of the entry tool used by the user and the entry area is equal to or less than a threshold value. Characteristic information presentation device. The information presentation apparatus according to claim 1 , wherein the reference information is a character string to be entered by the user in the entry area. The information presentation apparatus according to claim 2 , further comprising: a layout determination unit that determines a presentation layout of the reference information so that the layout is presented in a layout that fits in the entry area. Recognizing means for recognizing the character string that the user has already entered in the entry area with reference to the document image, and the size of the character string;
In the case where the layout is presented, the size of the character string corresponding to the recognized character string in the reference information is compared with the size of the recognized character string, and the comparison result is obtained. A notification means for notifying the user;
The information presentation apparatus according to claim 3 , further comprising: Recognizing means for recognizing characters already entered by the user in the entry area with reference to the document image,
The control means causes the presenting means to present the reference information so that a character string to be entered by the user can be distinguished from a character that has already been entered and a character that has not yet been entered. The information presentation apparatus according to claim 3 . Recognizing means for recognizing characters already entered in the entry area by referring to the document image;
A notification means for notifying a user when a character that the user has already entered in the entry area includes a character that the recognition means cannot recognize;
The information presentation apparatus according to claim 3 , further comprising: Said layout determination means may determine at least one of the line feed positions of character string to be presented according to the size and the reference information of a character to be presented according to the reference information, claims 3 to 6 The information presentation device according to any one of the above. From a document image obtained by photographing a second document containing the reference information, characterized by further comprising an extraction means for extracting the reference information, according to any one of claims 1 to 7 Information presentation device. It said presentation means is characterized by projecting the reference information in the vicinity of the entry region, the information presentation apparatus according to any one of claims 1 to 8. The presentation unit may be presented to a user by superimposing the reference information to the image including the first document, the information presentation apparatus according to any one of claims 1 to 2. The information presentation device according to any one of claims 1 to 10 ,
Photographing means for photographing the first document;
The presenting means;
An information presentation system comprising: An information presentation method performed by an information presentation device,
An image acquisition step of acquiring a document image obtained by photographing a first document including an entry area that can be filled in by the user so that the user also appears;
A specific step of specifying the position of the entry area;
An information acquisition step of acquiring reference information corresponding to the entry area;
Analyzing the document image to determine an arm used for entry by the user;
A control step of controlling the presenting means to present the reference information to the user so as to be visually recognized in the vicinity of the entry area according to the determination result in the arm determination step;
Equipped with a,
The first document includes a plurality of entry areas;
In the specifying step, an entry area that the user intends to fill out of the plurality of entry areas is specified,
In the information acquisition step, reference information corresponding to the specified entry area is acquired,
In the control step, the presenting means is controlled so that the reference information is presented to the user so as to be visually recognized in the vicinity of the specified entry area,
In the specifying step, when the distance between the tip of the writing tool used by the user and the writing area is equal to or less than a threshold value, the writing area is specified as the writing area that the user intends to fill in. A characteristic information presentation method. On the computer,
An image acquisition step of acquiring a document image obtained by photographing a first document including an entry area that can be filled in by the user so that the user also appears;
A specific step of specifying the position of the entry area;
An information acquisition step of acquiring reference information corresponding to the entry area;
Analyzing the document image to determine an arm used for entry by the user;
A control step of controlling the presenting means to present the reference information to the user so as to be visually recognized in the vicinity of the entry area according to the determination result in the arm determination step;
Was executed,
The first document includes a plurality of entry areas;
In the specifying step, an entry area that the user intends to fill out of the plurality of entry areas is specified,
In the information acquisition step, reference information corresponding to the specified entry area is acquired,
In the control step, the presenting means is controlled so that the reference information is presented to the user so as to be visually recognized in the vicinity of the specified entry area,
In the specifying step, when the distance between the tip of the writing tool used by the user for writing and the writing area is equal to or less than a threshold value, the program specifies the writing area as the writing area that the user intends to fill in .

Images