JP2017167810A - Input support device, input support method, control program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input support device for displaying in an intensified state, only an input portion in which an input has not been performed, in a document.SOLUTION: An input support device performs an input support to a document mounted on a calligraphy stage. The document has an input area on which an input is performed using a writing instrument. The input support device comprises: an image acquisition part 416 for acquiring an image in which a mount surface of the calligraphy stage on which the document is mounted, is imaged; an input area acquisition part 412 for acquiring an input area of the document from the image acquired by the image acquisition part 416; an input area recognition part 414 for recognizing whether or not the input has been performed to the acquired input area; a layout generation part 415 for generating an input support image for distinguishing an input area in which the input has been performed and an input area in which the input has not been performed; and a display part 406 for displaying the input support image on the calligraphy stage.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an input support apparatus that supports input to a document using a writing instrument or the like.

  There is an input support apparatus that provides input support to a user by emphasizing an input area of a paper document placed on a table. The input support device includes, for example, a mounting table on which a document is mounted, a camera, a projector, and a control device. Patent Document 1 discloses an input support apparatus having such a configuration. The camera images a predetermined range including a document placed on the placement surface of the placement table, and transmits the captured image to the control device. The control device determines the type from the image of the document included in the captured image, and creates information (hereinafter referred to as “work support information”) for highlighting the input area according to the type of the document. To the projector. The projector projects an image that highlights the input area of the document on the placement surface based on the work support information.

JP 2013-254437 A

  In the case of a document provided with a plurality of input areas, it may be placed on the mounting table while leaving an uninput input area. In this case, the input area that has already been input is also uniformly highlighted together with the input area that has not been input, making it difficult for the user to specify the input area to be input next. For this reason, it is desirable to highlight only an input area that has not been input in the document and highlight an input area that has already been input.

  In order to solve the above problem, it is a main object of the present invention to provide an input support apparatus that highlights only an input portion that has not been input in a document.

  The input support device according to the present invention is input by an image acquisition unit that acquires an image obtained by capturing a placement surface on which a document is placed, and the writing acquired by the document from the image acquired by the image acquisition unit. Input area acquisition means for acquiring an input area, input area recognition means for recognizing whether or not the acquired input area has been input, and input support for distinguishing between an input area that has been input and an input area that has not been input The image forming apparatus includes: a layout generation unit that generates an image; and a display unit that displays the input support image on the placement surface.

  According to the present invention, by displaying an input support image that distinguishes an input area that has not been input from an input area that has already been input, it is possible to highlight only an input area that has not been input in a document. Therefore, the user can easily recognize the non-input area.

1 is an overall configuration diagram of a system. (A)-(c) is explanatory drawing of an input assistance apparatus. The hardware configuration example figure of a controller part. (A), (b) is explanatory drawing of the function of an input assistance apparatus. (A), (b) is explanatory drawing of the document used as input assistance object. FIG. 5 is a view showing an example of a table showing a database for managing input areas. The flowchart of the process of a main control part. The flowchart of an input assistance process. (A)-(f) is an illustration figure of a highlight display. (A), (b) is an illustration figure of the table which shows the database which manages an input area. The flowchart of an input assistance process. (A), (b) is an illustration figure of highlight display. The flowchart of the process of a main control part. (A), (b) is an illustration figure of highlight display.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the present invention according to the claims, and all combinations of features described in the embodiments are not necessarily essential to the solution means of the present invention. .

[First Embodiment]
In this embodiment, an information processing apparatus incorporating a projector is used to effectively support input of documents such as slips, application forms, and notification forms in accordance with user input conditions. Hereinafter, an information processing apparatus incorporating a projector is referred to as an “input support apparatus”. Information representing an image (input support image) displayed by the projector to support the user is referred to as “input support information”.

  FIG. 1 is an overall configuration diagram of a system including an input support apparatus according to the present embodiment. This system includes an input support apparatus 101, a host computer 102, and a printer 103, which are communicably connected via a network 104 such as Ethernet (registered trademark). The network 104 is a WAN (Wide Area Network) such as a LAN (Local Area Network) or the Internet. This system can execute a scan function in which the input support apparatus 101 reads an image and a print function in which scan data is output by the printer 103 in accordance with an instruction from the host computer 102. It is also possible to execute the scan function and the print function by direct instructions to the input support apparatus 101 without using the host computer 102.

<Configuration of input support device>
FIG. 2 is an explanatory diagram of the input support apparatus 101. An overall configuration diagram of the input support apparatus 101 is shown in FIG. As illustrated, the input support apparatus 101 includes a controller unit 201, a camera unit 202, an arm unit 203, a short focus projector 207, and a distance image sensor unit 208. The controller unit 201, the camera unit 202, the projector 207, and the distance image sensor unit 208 are connected by an arm unit 203. The arm part 203 can be bent and stretched using a joint. The input support apparatus 101 includes a document table 204 as a mounting table on which the document 206 is mounted.

  The lenses of the camera unit 202 and the distance image sensor unit 208 are directed toward the document table 204, and can read an image in the reading area 205 surrounded by a broken line. In the illustrated example, the document 206 is placed in the reading area 205. Therefore, the input support apparatus 101 can read the document 206. The camera unit 202 may capture an image with a single resolution, but it is preferable that a high-resolution image capture and a low-resolution image capture are possible. The input support apparatus 101 may further include a touch panel and a speaker as a configuration.

  FIG. 2B is an explanatory diagram of a coordinate system in the input support apparatus 101. In the input support apparatus 101, a coordinate system such as a camera coordinate system, a distance image sensor coordinate system, and a projector coordinate system is defined for each hardware device. In these coordinate systems, an image plane captured by the RGB camera 363 of the camera unit 202 and the distance image sensor unit 208 or an image plane projected by the projector 207 is an XY plane. The direction orthogonal to the image plane is defined as the Z direction. Further, in order to be able to handle the three-dimensional data of these independent coordinate systems in a unified manner, an orthogonal coordinate system in which a plane including the document table 204 is an XY plane and a direction perpendicular to the XY plane is a Z axis. Define

  As an example of transforming the coordinate system, FIG. 2C illustrates a rectangular coordinate system, a space expressed using a camera coordinate system centered on the camera unit 202, and an image plane captured by the camera unit 202. Show the relationship. The three-dimensional point P [X, Y, Z] in the orthogonal coordinate system can be converted to the three-dimensional point Pc [Xc, Yc, Zc] in the camera coordinate system by (Equation 1).

  Here, Rc and tc are configured by external parameters determined by the posture (rotation) and position (translation) of the camera unit 202 with respect to the orthogonal coordinate system. Rc is called a 3 × 3 rotation matrix, and tc is called a translation vector. Conversely, a three-dimensional point defined in the camera coordinate system can be converted into an orthogonal coordinate system by (Equation 2).

  The two-dimensional camera image plane photographed by the camera unit 202 is obtained by converting the three-dimensional information in the three-dimensional space into two-dimensional information by the camera unit 202. That is, the three-dimensional point Pc [Xc, Yc, Zc] on the camera coordinate system is converted by perspective projection conversion to the two-dimensional coordinate pc [xp, yp] on the camera image plane according to (Equation 3). Can do.

  Here, A is a 3 × 3 matrix called an internal parameter of the camera and expressed by a focal length and an image center.

As described above, by using (Equation 1) and (Equation 3), the three-dimensional point group represented by the orthogonal coordinate system is converted into the three-dimensional point group coordinate or the camera image plane in the camera coordinate system. be able to.
It is assumed that the internal parameters of each hardware device and the position and orientation (external parameters) with respect to the orthogonal coordinate system are calibrated in advance by a known calibration method. Hereinafter, when there is no particular notice and it is expressed as a three-dimensional point group, it represents three-dimensional data in an orthogonal coordinate system.

<Configuration example of controller unit>
FIG. 3 is a diagram illustrating a hardware configuration example of the controller unit 201 which is the main body of the input support apparatus 101. The controller unit 201 is configured by connecting various functional components to the system bus 301. The functional components include a computer having a CPU 302, a RAM 303, a ROM 304, and an HDD 305 as basic components. A CPU (Central Processing Unit) 302 controls the overall operation of the controller unit 201 by executing a control program of the input support apparatus 101. That is, the control program is for operating the computer as the controller unit 201 or the input support apparatus 101.
A RAM (Random Access Memory) 303 is a volatile rewritable storage medium and functions as a work area of the CPU 302. A ROM (Read Only Memory) 304 is a non-rewritable storage medium, and stores a startup program for the CPU 302. The HDD 305 is a hard disk drive (HDD) having a larger capacity than the RAM 303. The HDD 305 stores a control program for the input support apparatus 101 that is executed by the controller unit 201. The HDD 305 may be another storage device having an equivalent function such as an SSD (Solid State Drive).

  Functional components connected to the system bus 301 include a network I / F (I / F is an abbreviation of an interface, the same applies hereinafter) 306, an image processor 307, and a camera I / F 308. Furthermore, a display controller 309, a serial I / F 310, an audio controller 311 and a USB (Universal Serial Bus) controller 312 are also included in the functional components.

When the input support apparatus 101 is activated when the power is turned on or the like, the CPU 302 executes the activation program stored in the ROM 304. This activation program is for reading a control program stored in the HDD 305 and developing it on the RAM 303. When executing the startup program, the CPU 302 executes the control program developed on the RAM 303 and performs control.
The CPU 302 also stores data used for the operation by the control program on the RAM 303 to read / write. Various settings necessary for operation by the control program and image data generated by camera input can be further stored on the HDD 305 and read / written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306.

The image processor 307 reads the image data stored in the RAM 303, performs predetermined image processing, and writes the processing result back to the RAM 303. Note that image processing executed by the image processor 307 includes rotation, scaling, color conversion, and the like.
The camera I / F 308 is connected to the camera unit 202 and the distance image sensor unit 208, and acquires image data from the camera unit 202 and distance image data from the distance image sensor unit 208 in accordance with an instruction from the CPU 302, and writes the acquired image data into the RAM 303. In addition, a control command from the CPU 302 is transmitted to the camera unit 202 and the distance image sensor unit 208 to set the camera unit 202 and the distance image sensor unit 208.

  The distance image sensor unit 208 is an infrared pattern projection type distance image sensor, and includes an infrared pattern projection unit 361, an infrared camera 362, and an RGB camera 363. The infrared pattern projection unit 361 projects a three-dimensional measurement pattern onto an object using infrared rays that are invisible to human eyes. The infrared camera 362 is a camera that captures an infrared three-dimensional measurement pattern projected on an object. The RGB camera 363 is a camera that captures visible light with RGB signals. In the first embodiment, an infrared pattern projection method is adopted as the distance image sensor unit 208, but a distance image sensor of another method may be used. For example, a stereo system that performs stereo stereoscopic vision with two RGB cameras, or a TOF (Time of Flight) system that measures distance by detecting the flight time of laser light may be used.

A display controller 309 controls display of an image on the display in accordance with an instruction from the CPU 302. The display controller 309 is connected to the short-focus projector 207 and the touch panel 330.
The serial I / F 310 inputs and outputs serial signals. In the present embodiment, the serial I / F 310 is connected to the touch panel 330. The CPU 302 acquires the coordinates (position) pressed via the serial I / F 310 when the touch panel 330 is pressed.

  The audio controller 311 is connected to the speaker 340. The audio controller 311 converts audio data into an analog audio signal in accordance with an instruction from the CPU 302 and outputs audio through the speaker 340. The USB controller 312 controls an external USB device in accordance with an instruction from the CPU 302. In this example, the USB controller 312 is connected to an external memory 350 such as a USB memory or an SD card, and reads / writes data from / to the external memory 350.

<Functional configuration>
The functions of the input support apparatus 101 will be described with reference to FIGS.
FIG. 4A is a configuration example diagram of the functional module group 401 formed in the controller unit 201 when the CPU 302 executes a control program. FIG. 4B is a sequence diagram showing the relationship between the functional modules. The control program is stored in the HDD 305 as described above, and is expanded on the RAM 303 when the CPU 302 is activated.
In the functional module group 401, the main control unit 402 is the center of control, and controls the controller unit 201 according to the procedure shown in FIG. In addition, the functional module group 401 includes an image acquisition unit 416, a recognition processing unit 417, a user interface unit 403, a display unit 406, a network communication unit 404, a data management unit 405, an input support unit 418, and the like. FIG. 5 is an explanatory diagram of a document 501 that is an input support target. FIG. 6 is an exemplary diagram of a table 600 showing a database for managing input areas of a document 501 that is an input support target. The input area is an area in which a user can input characters and the like with a writing instrument.

  The image acquisition unit 416 is a module that performs image input processing, and includes a camera image acquisition unit 407 and a distance image acquisition unit 408. The camera image acquisition unit 407 acquires an image output from the camera unit 202 via the camera I / F 308 and stores it in the RAM 303. The distance image acquisition unit 408 acquires a distance image (distance image data) output from the distance image sensor unit 208 via the camera I / F 308 and stores it in the RAM 303. The distance image data is generated based on the projection of the three-dimensional shape pattern from the infrared pattern projection unit 361, the RGB camera image by the RGB camera 363, and the infrared camera image obtained by photographing the three-dimensional shape pattern by the infrared camera 362.

The recognition processing unit 417 detects and recognizes an object such as a document on the document table 204 or a user's hand from images (image data) acquired by the camera image acquisition unit 407 and the distance image acquisition unit 408. For this purpose, the recognition processing unit 417 includes a gesture recognition unit 409 and an object detection unit 410.
The gesture recognition unit 409 recognizes the position of the hand on the document table 204 and the user's gesture such as touch from the acquired image. Specifically, the gesture recognition unit 409 continuously acquires images on the document table 204 from the image acquisition unit 416. The gesture recognizing unit 409 detects an outer shape from a three-dimensional point cloud and a skin color region of a hand obtained from continuously acquired images, and gestures such as a hand position and a touch on the document table 204 based on the detected outer diameter. Is detected. When the gesture recognition unit 409 detects the position and gesture of the hand, it notifies the main control unit 402 that the gesture has been detected (gesture detection notification).

The object detection unit 410 detects the timing at which an object is placed and stopped and the timing at which the object is removed. Specifically, the object detection unit 410 continuously acquires images on the document table 204 from the image acquisition unit 416, and the stationary timing when the object is placed on the document table 204 and the object is stationary and the object is the document table 204. The removal timing to be removed from is detected. The object detection unit 210 detects a stationary timing and a removal timing based on a difference value between a background image on the document table 204 and a camera frame image having a predetermined number of frames.
The removal timing is a timing at which the object is completely removed from the document table 204 and nothing is placed on the document table 204. If the object detection unit 410 determines that a stationary object is placed, the object detection unit 410 stores the last camera frame image. If the object detection unit 410 determines that the object is not a document having a predetermined thickness or less but a solid object having a predetermined thickness or more based on the distance image data, the object detection unit 410 adds the distance to the camera frame image. The image is converted into a three-dimensional point cloud and saved. The threshold value of the thickness of the object used for determining whether the object is a document or a three-dimensional object can be changed by setting.

  Further, the object detection unit 410 simultaneously generates object identification information for identifying whether the detected object is a document or a three-dimensional object other than a document at a stationary timing. The object identification information is information for identifying whether the object is a document or a three-dimensional object by determining whether or not the thickness of the object is equal to or less than a predetermined thickness based on the distance image, for example, the threshold value of the object thickness described above including. Here, the example in which the object detection unit 410 identifies the object only by the thickness of the object has been described, but it is determined whether or not the above-described region where the difference value of the camera frame image is a predetermined value is a rectangle. It may be added to the conditions. When the object detection unit 410 detects the stationary timing and the removal timing of the object placement described above, the notification that the object is detected (object detection notification), the notification that the removal of the object is detected (object removal detection notification), And the object identification information are sent to the main control unit 402. The object identification information sent to the main control unit 402 is stored in a data management unit 405 (to be described later) or a server connected via a network and is read out at an arbitrary timing.

  The user interface unit 403 receives a request from the main control unit 402 and generates GUI (Graphical User Interface) parts such as messages and buttons. The user interface unit 403 requests the display unit 406 to display the generated GUI component. The display unit 406 displays the requested GUI component on the projector 207 or the touch panel 330 via the display controller 309. The projector 207 is installed toward the document table 204. For this purpose, the GUI component is projected onto the document table 204.

  The main control unit 402 acquires a gesture operation such as a touch recognized by the gesture recognition unit 409, an input operation from the touch panel 330 via the serial I / F 310, and coordinates representing these touch positions. The user interface unit 403 determines the operation content (for example, a pressed button) by associating the content of the operation screen being drawn with the input coordinate. The user interface unit 403 notifies the main control unit 402 of this operation content. As a result, the main control unit 402 receives the operation of the operator.

  The network communication unit 404 communicates with other devices on the network 104 via a network I / F 306 using a predetermined communication protocol. The data management unit 405 stores and manages various data including work data generated during the operation of the functional module group 401 in a predetermined area on the HDD 305. The stored data is, for example, image data acquired by the image acquisition unit 416.

  The input support unit 418 executes an input support process related to an input operation for a document by a user. The input support unit 418 includes a document recognition unit 411, an input region acquisition unit 412, a document position detection unit 413, an input region recognition unit 414, and a layout generation unit 415. The input support unit 418 outputs the modules such as image data acquired by the camera image acquisition unit 407 and the distance image acquisition unit 408 and gesture information representing the gesture recognized by the gesture recognition unit 409 via the main control unit 402. Get the data.

  The document recognition unit 411 recognizes the type of document on the document table 204. Specifically, the document recognition unit 411 extracts feature points and feature amounts of the image acquired by the camera image acquisition unit 407, and features of a plurality of document images for each document type registered in advance in the database. Compare the similarity with points and features. Thereby, the document recognition unit 411 determines which document on the document table 204 matches the document on the database, and recognizes the type of the document. An extraction range of feature points and feature amounts of the image by the document recognition unit 411 is a reading area 205 imaged by the camera unit 202. The document recognition unit 411 may detect a difference between the camera image and the background image, specify a document area in the camera image, and extract only feature points and feature amounts of the document area. . A technique called DoG is generally used for extracting feature points, and a technique called SIFT or SURF is generally used for extracting feature quantities of each feature point.

  A database that associates the feature points and feature amounts of each document image is stored on the HDD 305 of the controller unit 201 or an external server provided on the network 104. When provided in an external server, the controller unit 201 accesses the external server via the network 104 and refers to a database that associates feature points and feature amounts of each document image stored in the external server. Document recognition may be performed by a method of recognizing a barcode or the like embedded in the document in advance, in addition to a method using feature points and feature amounts.

  The input area acquisition unit 412 acquires an area to be input by the user with a writing instrument such as a ballpoint pen based on information indicating the type of document recognized by the document recognition unit 411. The area to be input (hereinafter referred to as “input area”) is that the user actually has to manually input characters with a writing instrument in the application form such as the document 501 shown in FIG. It is an area. In the example of FIG. 5A, areas represented in gray are input areas 502 to 510 of the document 501. As shown in FIG. 5A, the input areas 502 to 510 are set in advance for each rectangular area surrounded by a frame line or the like. The input area is set by the administrator of the input support apparatus 101 for each document type using, for example, a dedicated application of the host computer 102 or a dedicated application of the input support apparatus 101. Information on the set input area is managed by a database that manages documents to be input supported. This database is stored in the HDD 305 and is represented by, for example, the table 600 shown in FIG. The input area acquisition unit 412 acquires the input areas 502 to 510 of the document 501 by referring to the database in FIG. 6 that associates the document 501 with the input areas 502 to 510.

  In the example of FIGS. 5A and 6, the input area 502 is associated with the row 601 of the table 600. The input area 503 is associated with the row 602 of the table 600. The input area 504 is associated with the row 603 of the table 600. Input area 505 is associated with row 604 of table 600. Input area 506 is associated with row 605 of table 600. Input area 507 is associated with row 606 of table 600. Input area 508 is associated with row 607 of table 600. Input area 509 is associated with row 608 of table 600. Input area 510 is associated with row 609 of table 600. In FIG. 6, the document ID (document ID) associated with the row 610 of the table 600 is different from the document ID associated with the rows 601 to 609. That is, an input area of a document different from the documents associated with the rows 601 to 609 is associated with the row 610. A document ID column 621 of the table 600 indicates a document ID (identification information) uniquely assigned to each document. For example, “1” is assigned as the document ID to the document 501 shown in FIG.

  A page column 622 of the table 600 indicates a page number where an input area in the document exists. The input area coordinates indicate the vertex coordinates of the diagonal of the rectangle in the document coordinate system (Xd, Yd, Zd) of each document, thereby designating the range of the input area as a rectangular area. An input column 623 of the table 600 indicates whether or not the input area of the document in the camera frame image has already been input by the user. When the input completed column 623 is “0”, it indicates a state in which there is no user input in the input area (hereinafter, the input area in this state is referred to as “non-input area”). When the input column 623 is “1”, it indicates a state in which the user input is already in the input area (hereinafter, the input area in this state is referred to as “input area”). Details of the input completion column 623 will be described later in the input area recognition unit 414.

  The document position detection unit 413 detects the position of the document placed on the placement surface of the document table 204 based on the camera image acquired by the camera image acquisition unit 407. For example, the position of the document can be expressed as the upper left vertex coordinates (Xd0, Yd0, 0) of the document in the orthogonal coordinate system of FIG. The input area coordinates of the document in the projector coordinate system and the camera coordinate system can be obtained by converting the upper left vertex coordinates (Xd0, Yd0, 0) of the document into the orthogonal coordinate system and further using the above-described conversion formula. The coordinate conversion method is as described above. In this embodiment, for simplicity, the thickness of the document is set to “0”, and the Zd component in the document coordinate system is not considered. However, by acquiring the distance image data by the distance image acquisition unit 408, it is possible to perform coordinate conversion into the projector coordinate system and the camera coordinate system in consideration of the thickness and distortion of the document. Since the conversion method considering the document thickness is a known technique (Japanese Patent Application No. 2014-095541), detailed description thereof is omitted here.

  Based on the input area and the document position acquired by the input area acquisition unit 412 and the document position detection unit 413, the input area recognition unit 414 adds the camera frame image document input area acquired by the camera image acquisition unit 407 to the input area of the document. Recognizes whether or not characters have already been input. Whether or not the input area has already been input is detected based on a difference value between the background image and the camera frame image. If there is an input in the input area, the input column 623 of the corresponding input area on the database in FIG. 6 is set to “1”, and if there is no input, the input column 623 is set to “0”. Whether or not there is already an input in the input area may be detected by character recognition processing using OCR (Optical character recognition) technology, and is not limited to a method of detecting based on the difference value.

  The layout generation unit 415 is operated by the projector 207 on the document table 204 based on the information of the input area, the document position, and the input completed column 623 by the input area acquisition unit 412, the document position detection unit 413, and the input region recognition unit 414. A layout image to be projected onto a document is generated. The layout image generated by the layout generation unit 415 highlights each input area of the document 501 in FIG. The layout generation unit 415 performs layout so that the input area determined to be input from the information in the input column 623 is highlighted in green or the like to indicate to the user that the input area has been input. Generate an image. The generated layout image is projected by the projector 207 by the display unit 406. Accordingly, the user can determine at a glance whether the input area is not input or has been input. The calculation of the highlight display position of the corresponding input area is performed by converting the coordinates of the input area into the projector coordinate system based on the document position. The transformation of the coordinate system is as described above. The color of the input area is not limited to red or green, and may be any color such as yellow or blue. In the present embodiment, the input area is green. However, the layout image may be generated so that only the non-input area is highlighted and the input area is not highlighted. Further, instead of filling the entire input area and displaying it, the input area may be highlighted with a frame. Further, instead of highlighting the input area using colors, the input area may be emphasized by displaying character information such as “a place where input is required” around the input area. Alternatively, a layout image may be generated in which the input area is indicated by a symbol such as an arrow mark and highlighted. In this way, the layout image is an input support image that can distinguish between an input area that has not been input and an input area that has already been input.

<Main control unit>
7 and 8 are flowcharts showing an outline of processing by the main control unit 402. FIG. FIG. 9 is an exemplary diagram of highlight display by the projector 207.

  When starting the processing, the main control unit 402 first performs initialization processing of the input support apparatus 101 (S701). The main control unit 402 performs initialization processing to confirm activation of each hardware, color adjustment of the camera image acquisition unit 407 and the distance image acquisition unit 408, correction of the distance sensor value, and nothing is placed on the document table 204. Perform background image acquisition in the absence.

  The main control unit 402 projects the UI screen of FIG. 9A on the document stage 204 by the projector 207 via the user interface unit 403 (S702). FIG. 9A shows a state where nothing is placed on the document table 204. At this time, the main control unit 402 projects a message 902 from the projector 207 to prompt the user to place a scan object on the document table 204. Further, the main control unit 402 projects an end icon 901 with the projector 207. When the user touches the end icon 901 on the document table 204, the gesture recognition unit 409 recognizes the touch operation on the end icon 901, and the processing of the main control unit 402 ends.

  The main control unit 402 determines whether or not there is an object detection notification by the object detection unit 410 (S703). The object detection notification by the object detection unit 410 is notified when the document 501 is placed on the document table 204 by the user, for example, as shown in FIG. FIG. 9B shows a state when the document 501 is placed on the placement surface of the document table 204 by the user. Note that the characters 904 and 905 in FIG. 9B represent characters previously input by the user in the document 501 and indicate a state in which characters are written in the input area 502 and the input area 503, respectively. .

  When there is an object detection notification (S703: Y), the main control unit 402 determines whether the object is a document or a three-dimensional object based on the object identification information of the object detected by the object detection unit 410. (S704). When the object is a three-dimensional object (S704: three-dimensional object), the main control unit 402 performs processing when the object is a three-dimensional object (S705). When the three-dimensional object is placed on the placement surface of the document table 204, for example, a warning message such as “the placed object is not a document” is projected onto the document table 204. Or you may project the message which suggests switching to another application.

  When the object is a document (S704: document), the main control unit 402 updates the message projected on the document stage 204 by the projector 207 to “document is being analyzed” via the user interface unit 403 (S706). . Specifically, as shown in FIG. 9B, when the document 501 is placed on the document table 204 and the document 501 is detected by the object detection unit 410, the projector 207 projects a message 903.

  The main control unit 402 performs input support processing by the input support unit 418 (S707). Details of the processing in S707 will be described later with reference to FIG. The main control unit 402 projects the layout image generated by the input support process onto the document stage 204 by the projector 207 via the user interface unit 403 (S708). The main control unit 402 simultaneously updates and displays the UI screen message. The projection display of the layout image generated by the input support process is, for example, a shaded area 907 and a shaded area 908 in FIG. 9C. FIG. 9C shows a state in which the input support unit 418 highlights the document 501. A shaded area 907 and a hatched area 908 in FIG. 9C show the highlight projection display by the projector 207. The shaded area 907 is red and the hatched area 908 is green, and the projector 207 projects the input area so as to fill it. The message is updated to a message 906 to prompt the user to input the document 501.

  The main control unit 402 determines whether or not there is an object removal detection notification by the object detection unit 410 (S709). When there is an object removal detection notification (S709: Y), the main control unit 402 determines that there is no object on the placement surface of the document table 204, and accordingly, FIG. The screen is re-projected by the projector 207 (S710). After re-projecting the screen, the main control unit 402 determines whether or not the user has touched and pressed the end icon 901 on the document table 204 (S711). Note that the main control unit 402 determines whether or not the end icon 901 is touch-pressed when there is no object detection notification (S703: N) or after the processing of S705. When the end icon 901 is touched and pressed (S711: Y), the main control unit 402 ends the process. Otherwise (S711: N), the main control unit 402 returns to the process of S703.

  When there is no object removal detection notification (S709: N), the main control unit 402 determines whether or not the user has touched and pressed the end icon 901 on the document table 204, similarly to the processing of S711 (S712). . When the end icon 901 is touched and pressed (S712: Y), the main control unit 402 ends the process.

  When the end icon 901 is not touched and pressed (S712: N), the main control unit 402 determines the presence or absence of a gesture detection notification by the gesture recognition unit 409 (S713). The gesture recognizing unit 409 detects the user's gesture at the time of FIG. 9D, for example, and notifies the main control unit 402 of a gesture detection notification. FIG. 9D shows a state in which the user is inputting to the document 501 using a writing instrument (for example, a ballpoint pen). A hand 909 in FIG. 9D shows the user's left hand and right hand, and represents a state in which a character 501 is pressed with the left hand and characters are input into the input area 504 with a writing instrument held in the right hand. When there is no gesture detection notification (S713: Y), the main control unit 402 returns to the process of S709.

  When there is a gesture detection notification (S713: Y), the main control unit 402 determines whether or not the detected gesture is a gesture related to document input (S714). The gesture related to document input is determined by whether or not the user's hand has touched the input area of the document 501 on the document table 204 for a predetermined time or more. For example, when the user's hand 909 in FIG. 9D is placed in the input areas 504 and 505 of the document 501 for 3 seconds or more, it is determined that the user's hand is touching for a predetermined time or more. However, the predetermined time can be changed by setting and is not limited to 3 seconds. In this embodiment, the input is determined when the hand 909 has touched the input area for a predetermined time or more. However, when the hand 909 overlaps the area of the document 501, the gesture is related to document input. You may judge. When the gesture is not related to document input (S714: N), the main control unit 402 returns to the process of S709.

  When the gesture is related to document input (S714: Y), the main control unit 402 determines whether or not the user's gesture detected in S713 has ended (S715). The state in which the gesture has ended is specifically a state in which the user's hand 909 has disappeared from the document table 204. If the gesture has ended (S715: Y), the main control unit 402 returns to the process of S706. If the gesture has not ended (S715: Y), the main control unit 402 loops the process of S715 until the gesture ends. The reason for determining the end of the gesture in the process of S715 is to acquire a camera frame image again and perform document recognition in the input support process of S707. If the input support process is started in a state where the gesture has not ended, a hand may appear in the camera frame image, and document recognition may not be performed correctly, and document recognition may not be performed accurately.

  FIGS. 9E and 9F show the state on the document table 204 after the gesture as shown in FIG. 9D ends, that is, after the process of S715. FIG. 9 (e) shows the state on the document table 204 when the hand 909 is released from the document 501 after the user inputs characters in FIG. 9 (d), and the UI screen is updated by the processing of S706. This is the display when A character 911 in FIG. 9E is a character that can be seen as a result of input by the user on the screen in FIG. 9D. FIG. 9F shows a state when the document 501 in FIG. 9E is highlighted again by the processing of S707 and S708 by the main control unit 402 and the input support unit 418. A shaded area 913 and a shaded area 914 in FIG. 9F show the highlight display by the projector 207. In FIG. 9 (f), unlike the shaded range 907 and the shaded range 908 in FIG. 9 (c), the shaded range indicating non-input by the input area 504 is reduced by the input of the character 911 by the user. This indicates that the shaded area indicating completion has increased.

  The processes in S713 to S715 are processes for automatically executing the input support process in S707 without bothering the user according to the change in the state of the document 501. The state change of the document 501 includes, for example, that the document 501 is input with a ballpoint pen or the like, the position of the document is moved, and the like. Accordingly, it is assumed that the state of the document 501 has changed by detecting that the position coordinate of the document 501 has changed based on the camera frame image, instead of detecting the state change based on the gesture in the processing of S713 to S715. You may judge. Further, instead of automatically executing the input support processing in S707, the user may touch and press an input support processing execution button (not shown) projected and displayed on the document table 204 to execute it manually. Good.

<Input support processing>
FIG. 8 is a flowchart showing the input support process of S707. FIG. 10 is an example of a table showing a database that manages the input area of the document 501.

  The main control unit 402 acquires a camera frame image from the camera image acquisition unit 407 (S801). For example, the main control unit 402 acquires the screen of FIG. 9B as a camera frame image for performing subsequent processing. The main control unit 402 causes the document recognition unit 411 to recognize the document 501 on the document table 204 based on the acquired camera frame image (S802). The recognition process of the document recognition unit 411 is as described above.

  The main control unit 402 determines the similarity between the image of the document 501 on the document table 204 and the document image registered in the database by the document recognition unit 411, and determines whether the document 501 is an input support target ( S803). When the document 501 is an input support target (S803: Y), the main control unit 402 acquires the input area of the document 501 by the input area acquisition unit 412 (S804). The input area acquisition unit 412 acquires, for example, the input areas 502 to 510 in FIG. 9B. The processing of the input area acquisition unit 412 is as described above. The main control unit 402 uses the document position detection unit 413 to detect the position of the document on the placement surface of the document table 204 based on the camera frame image (S805). The processing of the document position detection unit 413 is as described above.

  Based on the input region of the document 501 acquired by the input region recognition unit 414 and the document position detected by the document position detection unit 413, the main control unit 402 adds characters or the like to the input region of the document on the camera frame image. It is recognized whether there is an input (S806). For example, “JAPAN” (character 904) and “Hakusan” (character 905) are input to the input area 502 and the input area 503 in FIG. At this time, the input area recognition unit 414 recognizes that the characters 904 and 905 are input to the input area 502 and the input area 503. Thereby, the input area recognition unit 414 determines that the input area 502 and the input area 503 have been input. In accordance with this determination result, the database of FIG. 6 is updated as shown in FIG. Specifically, the input columns 623 in the rows 601 and 602 in FIG. 6 indicating the information of the input area 502 and the input area 503 have been input as the input columns 651 in the rows 1001 and 1002 in FIG. It is updated to “1” indicating that The processing of the input area recognition unit 414 is as described above.

  After updating the database, the main controller 402 causes the layout generator 415 to generate a highlight display layout image based on the document position and the database updated in S806 (S807). The main control unit 402 generates a layout image for highlight display even when the document 501 is not an input support target (S803: Y). According to the database in FIG. 10A, it is determined that the input area 502 and the input area 503 of the document 501 are already input based on the lines 1001 and 1002. For this purpose, the layout generation unit 415 generates a layout image that highlights the input areas 502 and 503 in green and the other input areas 504 to 510 in red. Specifically, the layout generation unit 415 generates a layout image to be highlighted as in the shaded area 907 and the shaded area 908 in FIG. The processing of the layout generation unit 415 is as described above. When a character 911 is additionally written in the input area 504 as shown in FIG. 9E, the database row 1003 is updated as shown in FIG. 10B by the processing of S806 described above. At this time, the layout image for highlight display is as shown by the shaded area 913 and the hatched area 914 in FIG.

  As described above, the input support apparatus 101 according to the present embodiment projects a screen for input support in accordance with the input status of the document 501 placed on the placement surface of the document table 204 by the user. Therefore, the user can appropriately grasp the place to be input in the document. In addition, the input support apparatus 101 can update the display for input support at any time without bothering the user in order to determine the input work state of the user and automatically determine the input status. it can.

[Second Embodiment]
The input support apparatus according to the second embodiment clearly indicates to the user that omissions in document input occur according to the progress of document input. Since the hardware configuration and functional blocks of the input support apparatus 101 are the same as those in the first embodiment, description thereof is omitted.

  FIG. 11 is a flowchart of the input support process (S707) of FIG. 7 according to the second embodiment. The processes before and after the input support process are the same as those in the first embodiment shown in FIG. FIG. 12 is a view showing an example of highlight display by the projector 207.

When the input support process is started, the main control unit 402 recognizes whether there is an input of characters or the like in the input area of the document by the same process as S801 to S806 in FIG. 8 (S801 to S806).
The main control unit 402 uses the input support unit 418 to determine the input status of the document by the user (S1101). Specifically, the input support unit 418 determines that two input areas of the document 501 have been input based on the rows 1001 and 1002 of the database in FIG. Since the document 501 has nine input areas 502 to 510, the input support unit 418 determines that the input state of two (9/9) input areas as a whole has been input based on the database. Determine.

  The main control unit 402 causes the layout generation unit 415 to generate a layout image for highlight display based on the determination result of the document position, the database, and the input status (S1102). Specifically, the layout generation unit 415 adds an input status determination result such as the message 1201 in FIG. 12A to the layout image generated in the first embodiment, and creates a layout image for highlight display. Generate. Similarly, in FIG. 12B, a message 1202 representing the determination result of the input status based on the database of FIG. 10B is added to the layout image generated in the first embodiment, and the layout image for highlight display is displayed. Is generated. In the example of FIG. 12, the input status is displayed as a fraction, but this may be displayed as a percentage, and the method of expressing how far the entire input area has been input is not limited to this.

  As described above, in the second embodiment, in addition to the effects of the first embodiment, it is easy to recognize how much the user has input to the document as a whole by using the input status determination processing in S1101. Therefore, the user can grasp the rough work time required for input. In addition, this input support apparatus clearly indicates how far the input has been completed and how many non-input areas are present as numerical values, so that input leakage can be prevented in advance.

[Third Embodiment]
The input support apparatus according to the third embodiment dynamically controls the display of the projector 207 so as not to interfere with input when the user is inputting a document. Since the hardware configuration and functional blocks of the input support apparatus 101 are the same as those in the first embodiment, description thereof is omitted.

  FIG. 13 is a flowchart showing an outline of processing by the main control unit 402. FIG. 14 is a view showing an example of highlight display by the projector 207. The processes in S701 to S713 and S715 in FIG. 13 are the same as S701 to S713 and S715 in FIG.

  When there is a gesture detection notification by the gesture recognizing unit 409 (S713: Y), the main control unit 402 has the user's hand 909 in the vicinity of the input areas 502 to 510 of the document 501 as illustrated in FIG. It is determined whether or not (S1301). The determination as to whether or not the area is in the vicinity of the input areas 502 to 510 is made, for example, when the shortest distance between the outline of the hand 909 and the outer frame line of each input area 502 to 510 on the document 501 is within a predetermined distance, for example 3 [ cm] or less. The contour line of the hand 909 is the contour line of the region of the hand 909 that appears in the camera frame image. Whether or not the shortest distance is within 3 [cm] is determined by analyzing a camera frame image. For example, in FIG. 14A, an area near the hand 909 is determined by calculating the shortest distance between the contour line of the hand 909 and each of the input areas 502-510. It is determined that the input areas 502 to 505 having the shortest distance within 3 [cm] are in the vicinity of the hand 909. However, the value used as a criterion for determining whether or not the object is in the vicinity can be changed, and is not limited to 3 [cm].

  When the hand 909 is not in the vicinity of any of the input areas 502 to 510 (S1301: N), the main control unit 402 repeatedly performs the processes after S709. When the hand 909 is in the vicinity of any one of the input areas 502 to 510 (S1301: Y), the main control unit 402 performs projection display by the projector 207 according to the distance between the hand 909 and each of the input areas 502 to 510. Is changed (S1302). The main control unit 402 changes the display by the projector 207 to leave the highlight display of the shaded area 1402 while highlighting the input areas 502 to 505 determined to be in the vicinity of the hand 909. Clear the display. That is, the input support image is changed so that the display in the input areas 502 to 505 determined to be in the vicinity of the hand 909 becomes inconspicuous. Thereby, the user can input the document 501 without being interrupted by the projection light of the projector 207. Further, as shown by a frame line 1403 in FIG. 14B, the highlighted area that has been painted and displayed may be changed so that only the frame line remains. Accordingly, it is possible to reduce the hindrance to the input work due to the projection light of the projector 207 while showing the input location to the user.

  As described above, in the third embodiment, in addition to the effects of the first embodiment, it is possible to reduce the hindrance of input work due to the projection of the projector 207 by determining the vicinity of the hand and changing the display by the projector 207. Moreover, you may combine 3rd Embodiment and 2nd Embodiment.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

  DESCRIPTION OF SYMBOLS 101 ... Input support device 201 ... Controller part 202 ... Camera part 204 ... Operation plane 207 ... Projector 208 ... Distance image sensor part 330 ... Touch panel 402 ... Main control part 409 ... Gesture recognition part 420 ... Touch panel event detector

Claims (11)

Image acquisition means for acquiring an image obtained by imaging a placement surface on which a document is placed;
From the image acquired by the image acquisition means, the document has an input area acquisition means for acquiring an input area input by a writing instrument,
Input area recognition means for recognizing whether or not the acquired input area has been input;
Layout generating means for generating an input support image that distinguishes an input area that has been input from an input area that has not been input;
Display means for displaying the input support image on the placement surface, characterized in that,
Input support device. Based on the image acquired by the image acquisition means, comprising a document position detection means for detecting the position of the document placed on the placement surface,
The display means displays the input support image at a position corresponding to the position of the document on the placement surface detected by the document position detection means.
The input support apparatus according to claim 1. Registration means for registering a plurality of document images for each document type;
Storage means for storing information on the input area set for each type of the document;
A document recognition unit that recognizes the type of the document placed on the placement surface by comparing the plurality of document images registered in the registration unit and the image acquired by the image acquisition unit; With
The input area acquisition means refers to the storage means, and acquires the input area according to the type of the document recognized by the document recognition means.
The input support apparatus according to claim 1 or 2. The input area recognizing means, based on the input area acquired by the input area acquiring means and the position of the document detected by the document position detecting means, of the document included in the image acquired by the image acquiring means. Recognizing whether or not the input area has been input,
The input support apparatus according to claim 2 or 3. Based on the image continuously acquired by the image acquisition means, comprising a gesture recognition means for recognizing a user's gesture,
When the gesture is related to input, the layout generation unit updates the input support image after the gesture ends,
The display means displays the updated input support image on the placement surface,
The input support device according to claim 1. An input support means for determining an input status to the document based on the number of the input areas of the document and the number of input areas already input;
The layout generation means generates the input support image including a determination result of the input situation,
The input support device according to claim 1. Based on the image acquired by the image acquisition means, comprising a determination means for determining whether the input area and the user's hand are within a predetermined distance;
The layout generation means is configured to provide the input support so that an input area whose distance from the user's hand is within the predetermined distance becomes inconspicuous when the input area and the user's hand are within the predetermined distance. Characterized by changing the image,
The input support apparatus according to claim 1. The display means is a projector that projects the input support image onto the position of the document on the placement surface.
The input support apparatus according to claim 1. A method executed by an information processing apparatus comprising: an imaging unit that images a placement surface on which a document is placed; and a display unit that displays an image on the placement surface.
From the image captured by the imaging means, obtain an input area that is input by the writing instrument of the document,
Recognizing whether the acquired input area has been input,
Generate an input support image that distinguishes input areas that have already been input from input areas that have not been input,
The generated input support image is displayed on the placement surface by the display means,
Input support method. A computer comprising an imaging means for imaging a placement surface on which a document is placed;
Image acquisition means for acquiring an image captured by the imaging means;
An input area acquisition means for acquiring an input area input by a writing instrument from the image acquired by the image acquisition means;
Input area recognition means for recognizing whether or not the acquired input area has been input;
Layout generating means for generating an input support image for distinguishing an input area that has been input from an input area that has not been input;
Display means for displaying the input support image on the placement surface;
Control program to function as   A computer-readable storage medium storing the control program according to claim 10.