JP5999236B2 - INFORMATION PROCESSING SYSTEM, ITS CONTROL METHOD, AND PROGRAM, AND INFORMATION PROCESSING DEVICE, ITS CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to an information processing system capable of superimposing a locus of a touch operation on projection data projected by a projection device using a mobile terminal including a touch panel, a control method thereof, a program, and an information processing device, The present invention relates to a control method and a program.

  Conventionally, when consenting to the contents of a paper document such as a contract, the paper document is signed by handwriting. However, each company is making paperless from the viewpoint of efficiency of operations and cost reduction. Therefore, a method of electronically signing a paper document to be signed without printing is considered.

  For example, the following Patent Document 1 discloses a mechanism for attaching user handwritten signature image data generated by handwriting input to document data together with the user's electronic signature and electronic certificate.

JP 2003-263647 A

  By the way, for example, in a bank product explanation service, a teller who is a bank explainer explains insurance products and financial products to customers. In this description, there is a case in which a screen for product explanation is projected on a desk by a projector, and the screen display is switched in response to a touch operation from the user on the projected screen.

  This mechanism also requires paperless signatures when customers make contracts with banks. In view of this, a method has been considered in which a customer performs a signature by touching a screen projected by a projector. That is, it is a method of detecting the locus of the touch operation and using the locus as a signature.

  However, depending on the performance of the infrared sensor that detects the touch operation, there is a problem that fine movement of the touch operation cannot be detected and the locus cannot be acquired well. Therefore, a method of accepting a signature using a mobile terminal equipped with a touch panel is conceivable, but an operation for designating a transmission destination of the signature or instructing transmission of the signature is necessary, which is troublesome for the user.

  It is an object of the present invention to provide a mechanism that can superimpose a trajectory of a touch operation on projection data projected by a projection device using a portable terminal that includes a touch panel.

  In order to achieve the above object, an information processing system of the present invention is an information processing system including a projection device that projects projection data, a mobile terminal that acquires a trajectory of a touch operation on a touch panel, and an information processing device. The information processing apparatus is configured to store a position information indicating a position in the projection data, and a position in the projection data projected by the projection apparatus at a position indicated by the position information stored in the storage means. Trajectory acquisition means for acquiring information obtained from the trajectory of the touch operation from the placed mobile terminal, and information obtained from the trajectory acquired by the trajectory acquisition means at the position in the projection data indicated by the position information And superimposing means for superimposing.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to superimpose the locus | trajectory of touch operation on the projection data projected by a projector using a portable terminal provided with a touch panel.

It is a figure which shows an example of the system configuration | structure of the information processing system in embodiment of this invention. It is a figure which shows an example of the outline | summary of the external appearance structure and internal structure of the projection imaging device. It is a figure which shows an example in the case of demonstrating using the projection image projected using the projection imaging device. It is a figure which shows an example of the hardware constitutions of the information processing apparatus 101 and the projection imaging apparatus 102. 3 is a diagram illustrating an example of a hardware configuration of a mobile terminal 103. FIG. 2 is a diagram illustrating an example of functional configurations of an information processing apparatus 101 and a mobile terminal 103. FIG. FIG. 10 is a diagram illustrating an example of a flowchart of processing for accepting an operation on a menu screen. 5 is a diagram illustrating an example of a screen configuration of a menu screen 800. FIG. It is a figure which shows an example of the flowchart of document entry processing. It is a figure which shows an example of a screen structure of the document entry screen. 5 is a diagram illustrating an example of a table configuration of a document data management table 1100, a portable terminal management table 1110, and a signed document management table 1120. FIG. 6 is a diagram showing a state in which a mobile terminal 103 is arranged at a position of a signature field area 1002. FIG. 6 is a diagram showing an example of a screen configuration of a signature input screen 1300. FIG. 6 is a diagram illustrating a state in which the mobile terminal 103 has moved from the position of a signature field area 1002. FIG. It is a figure which shows an example of the flowchart of document entry processing in 2nd Embodiment. It is a figure which shows the outline | summary of the document entry process in 2nd Embodiment.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to an embodiment (first embodiment) of the present invention. The information processing system includes an information processing apparatus 101, a projection imaging apparatus 102, and a portable terminal 103 (transmission destination terminal). The information processing apparatus 101 is communicably connected to the projection imaging apparatus 102, and is further connected to the display 410 and the input device 409 so as to be communicable. In FIG. 1, these devices are connected by wire, but may be wireless. The information processing apparatus 101 and the portable terminal 103 are connected to the same network, and can communicate with each other via the network. The configuration of various terminals connected on the system of FIG. 1 is an example, and there are various configuration examples depending on the application and purpose.

  The information processing apparatus 101 is a general-purpose apparatus equipped with an operating system. The information processing apparatus 101 instructs various devices connected to the information processing apparatus 101 to execute various operations.

  The projection imaging apparatus 102 is an apparatus that includes a digital camera, an infrared sensor, and a projector. Projection is performed by a projector in accordance with an instruction from the information processing apparatus 101, the projected projection surface such as a desk is imaged by a digital camera, and the movement of the user's hand and the position of the object are recognized on the projection surface.

  An example of the outline of the external structure and internal structure of the projection imaging apparatus 102 will be described with reference to FIG.

  FIG. 2 is a front view and a right side view of the projection imaging apparatus 102. The hidden lines of the internal structure are represented by broken lines. The projection imaging apparatus 102 includes a storage unit 203 that stores a digital camera 421 (imaging apparatus), an infrared sensor 422 (sensor apparatus), and a projector 423 (projection apparatus). The storage unit 203 is fixed by being sandwiched from the left and right by the bottom surface 201, the left side surface 202a, and the right side surface 202b.

  An upper surface 204 is installed above the left side surface 202a and the right side surface 202b, and a lower surface of the upper surface 204 is provided with a reflecting mirror 207 for reflecting light emitted from the projector 423. In other words, the projector 423 is installed so that it can project toward the reflecting mirror 207 through the first opening 205. Similarly, the digital camera 421 is installed toward the reflecting mirror 207 through the second opening 206 in order to capture an image of the projection surface reflected by the reflecting mirror 207.

  The infrared sensor 422 is a depth sensor using infrared rays, for example. This is to calculate the distance from the infrared sensor 422 to these objects in order to specify the position and movement of the user's hand, paper document (hereinafter referred to as document), etc. on the projection surface projected by the projector 423. The infrared sensor 422 calculates a distance to an object by projecting a near-infrared dot pattern onto the real space and acquiring the shape and interval of the dots. Alternatively, the distance may be calculated by projecting infrared light, reading the infrared light reflected by a camera included in the infrared sensor 422, and calculating the time when the infrared light reciprocated. As long as the position and movement of the object in the real space can be acquired, any method may be used for calculating the distance of the infrared sensor 422, and a sensor other than the infrared sensor 422 may be used. Similarly to the digital camera 421, the infrared sensor 422 is also installed toward the reflecting mirror 207 through the second opening 206. Infrared rays are irradiated toward the reflecting mirror 207.

  As described above, the projection imaging apparatus 102 includes three apparatuses, which are connected to the information processing apparatus 101 so as to communicate with each other. In the present embodiment, the information processing apparatus 101 and the projection imaging apparatus 102 are different casings, but they may be the same casing. That is, the information processing apparatus 101 may be further included in the projection imaging apparatus 102.

  Returning to FIG. The mobile terminal 103 is a mobile terminal including a touch panel such as a smartphone or a tablet terminal. The portable terminal 103 can acquire the locus of touch operations on the touch panel. In the present embodiment, the portable terminal 103 is connected to the information processing apparatus 101 in advance so as to communicate with each other by wireless connection. Note that the connection between the portable terminal 103 and the information processing apparatus 101 may be a wired connection.

  Next, the positional relationship between the projection imaging apparatus 102 and the user in this embodiment will be described with reference to FIG. The projection imaging apparatus 102 is installed on the desk 303. In this embodiment, there are two users across the desk. A teller 301 explaining at a bank window and a customer 302 receiving the explanation.

  When the teller 301 operates the input device 409 to instruct the information processing apparatus 101 to display a screen, the information processing apparatus 101 transmits a projection instruction for the screen to the projector 423. Upon receiving this, the projector 423 projects a screen with the left and right reversed toward the reflecting mirror 207. This is because the image projected on the projection surface 304 is reversed left and right by being reflected by the reflecting mirror 207. As a result, the designated screen is projected onto the projection plane 304. The process of inverting the screen left and right may be performed by the information processing apparatus 101 or the projector 423. The projection surface 304 is preferably a white surface without a pattern so that the projected light can be easily seen.

  When the light emitted from the projector 423 is reflected by the reflecting mirror 207 in the direction indicated by the dashed arrow in FIG. 3, the light is projected onto the projection plane 304. The teller 301 explains to the customer 302 by looking at the screen expressed by this light. In the present embodiment, the description will be made on the assumption that the teller 301 and the customer 302 are positioned with the desk interposed therebetween and are operated while viewing the projected screen.

  FIG. 4 is a diagram illustrating an example of the hardware configuration of the information processing apparatus 101 and the projection imaging apparatus 102.

  The CPU 401 comprehensively controls each device and controller connected to the system bus 404.

  Further, the ROM 402 or the external memory 411 (storage means) realizes a BIOS (Basic Input / Output System), an operating system program (hereinafter referred to as OS), which is a control program of the CPU 401, and a function executed by the information processing apparatus 101. Various programs and the like necessary to do this are stored. The RAM 403 functions as a main memory, work area, and the like for the CPU 401.

  The CPU 401 implements various operations by loading a program or the like necessary for execution of processing into the RAM 403 and executing the program.

  The general-purpose input / output controller 405 controls input / output with an input device 409 such as a keyboard and a mouse, and with the digital camera 421 and the infrared sensor 422 of the projection imaging apparatus 102. These are connected according to a bus standard such as USB (Universal Serial Bus).

  The video controller 406 controls display on a display device such as the display 410 and the projector 423 of the projection imaging device 102 and a projection device. The display 410 is, for example, a CRT or a liquid crystal display. Further, the video controller 406 instructs the display 410 and the projector 423 of the projection imaging apparatus 102 to display and project the same screen or different screens. These can be so-called mirroring or multi-monitor.

  The memory controller 407 controls access to a hard disk that stores a boot program, browser software, various applications, font data, user files, editing files, various data, and the like. The memory controller 407 also controls access to an external memory 411 such as a card-type memory connected to the flexible disk or PCMCIA card slot via an adapter.

  The communication I / F controller 408 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, Internet communication using TCP / IP is possible.

  Note that the CPU 401 can perform display on the display 410 and projection by the projector 423 of the projection imaging apparatus 102 by executing outline font rasterization processing on a display information area in the RAM 403, for example. In addition, the CPU 401 allows a user instruction with a mouse cursor (not shown) on the display 410. Further, the CPU 401 enables a user instruction by touching the screen projected on the projection plane 304 with the user's hand. That is, according to the coordinate position of the user's hand in the real space detected by the infrared sensor 422, it is detected whether or not a button or the like on the screen is pressed, and the CPU 401 is caused to execute an operation.

  Various programs used by the information processing apparatus 101 of the present invention to execute various processes to be described later are recorded in the external memory 411 and are executed by the CPU 401 by being loaded into the RAM 403 as necessary. is there. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 411.

  FIG. 5 is a diagram illustrating an example of a hardware configuration of the mobile terminal 103.

  The CPU 501 comprehensively controls each device and controller connected to the system bus 504.

  The ROM 502 or flash memory 514 stores a BIOS and OS that are control programs for the CPU 501, and various programs described below that are necessary for realizing the functions executed by the mobile terminal 103. The RAM 503 functions as a main memory, work area, and the like for the CPU 501.

  The CPU 501 implements various operations by loading a program or the like necessary for execution of processing into the RAM 503 and executing the program.

  The input controller 505 controls input from the touch panel 510, the microphone 511, and the camera 512. The touch panel 510 controls input related to the user's touch operation, the microphone 511 controls input of audio, and the camera 512 controls input of captured still images and moving images. The output controller 506 controls output to the touch panel 510 and the speaker 513.

  The touch panel 510 detects a touch operation from the user and displays an image sent from the output controller 506 described above. The touch panel 510 is a component in which a display and a position input device are integrated. Touch operations (hereinafter referred to as multi-touch) for a plurality of locations can also be detected.

  The memory controller 507 controls access to the flash memory 514 that stores a boot program, browser software, various applications, font data, user files, editing files, various data, and the like. In this embodiment, the flash memory is described, but a storage medium such as a hard disk, a flexible disk, or a card-type memory connected to a PCMCIA card slot via an adapter may be used.

  The communication I / F controller 509 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, Internet communication using TCP / IP is possible.

  The sensor controller 508 controls input from the sensor 515 included in the mobile terminal 103. Various sensors exist in the sensor 515 of the mobile terminal 103, such as an orientation sensor and an acceleration sensor.

  Note that the CPU 501 enables display on the touch panel 510 by executing outline font rasterization processing on a display information area in the RAM 503, for example.

  Various programs and the like used for the portable terminal 103 of the present invention to execute various processes to be described later are recorded in the flash memory 514 and are executed by the CPU 501 by being loaded into the RAM 503 as necessary. . Further, definition files and various information tables used by the program according to the present invention are stored in the flash memory 514.

  Next, an example of a functional configuration of the information processing apparatus 101 and the mobile terminal 103 will be described with reference to FIG. Note that the functional configurations of the information processing apparatus 101 and the mobile terminal 103 in FIG. 6 are merely examples, and there are various configuration examples depending on applications and purposes.

  The information processing apparatus 101 includes, as functional units, a storage unit 601, a display control unit 602, a projection control unit 603, an imaging control unit 604, a sensor control unit 605, a coordinate conversion unit 606, an operation detection unit 607, A communication control unit 608 and a signature superimposing unit 609 are provided.

  The storage unit 601 is a functional unit for storing information of various tables to be described later and information constituting various screens. Add, update, and delete information as necessary.

  The display control unit 602 is a functional unit for controlling the display on the display 410 that is communicably connected to the information processing apparatus 101. When a display instruction such as a screen is issued from the display control unit 602, the display instruction is transmitted to the display 410 via the video controller 406. Upon receiving this instruction, the display 410 displays a screen or the like according to the instruction.

  The projection control unit 603 is a functional unit for controlling the projection of the projector 423 that is communicably connected to the information processing apparatus 101. When a projection instruction such as a screen is issued from the projection control unit 603, the projection instruction is transmitted to the projector 423 via the video controller 406 as in the display control unit 602. Upon receiving this instruction, the projector 423 projects a screen or the like on the real space according to the instruction. In this embodiment, the case where document data representing a document is projected will be described as an example. However, the data to be projected (projection data) is not limited to document data, and may be image data or text data. Also good. Or the screen which combined these may be sufficient.

  The imaging control unit 604 is a functional unit for controlling imaging of the digital camera 421 connected to the information processing apparatus 101 so as to be communicable. When an imaging instruction for the projection plane 304 is issued from the imaging control unit 604, an imaging instruction is issued to the digital camera 421 via the general-purpose input / output controller 405. Upon receiving this instruction, the digital camera 421 performs imaging and transmits image data generated by the imaging to the information processing apparatus 101. The imaging control unit 604 receives this, reverses the image data left and right, and then the storage unit 601 stores it in the external memory 411 or the like. As described above, since the digital camera 421 captures an image taken on the reflecting mirror 207, the left and right sides are reversed. In order to correct this, before the image data is stored in the storage unit 601, the imaging control unit 604 inverts the image data left and right.

  The sensor control unit 605 receives information on the real space (such as the presence / absence and position of an object) on the projection plane 304 transmitted from the infrared sensor 422 that is communicably connected to the information processing apparatus 101, and determines the state of the real space. It is a functional part for detecting. Information on the real space detected by the infrared sensor 422 is transmitted from the infrared sensor 422 to the information processing apparatus 101 one by one. The sensor control unit 605 receives this and passes it to a coordinate conversion unit 606 described later as necessary.

  The coordinate conversion unit 606 is a functional unit for converting the coordinate value indicating the position of the object on the projection plane 304 included in the real space information received by the sensor control unit 605. The coordinate conversion unit 606 converts this coordinate value into a coordinate value on the screen projected on the projection plane 304. That is, the coordinate conversion unit 606 is used to specify which coordinate value on the screen managed by the information processing apparatus 101 corresponds to the coordinate value detected by the infrared sensor 422. Note that the coordinate conversion method uses conventional techniques, and thus detailed description thereof is omitted.

  The operation detection unit 607 is a functional unit for detecting an operation from the user. An instruction from the user transmitted through the input device 409 connected to the information processing apparatus 101 is detected, or an instruction given by the user on the screen projected on the projection surface 304 is detected. The latter instruction is detected by detecting which button on the screen is pressed by converting the coordinate value of the object detected on the projection plane 304 by the coordinate conversion unit 606 or by placing the document. Detect whether capture is instructed.

  The communication control unit 608 is a functional unit for controlling communication between the information processing apparatus 101 and the mobile terminal 103. The communication control unit 608 controls transmission / reception of information between the information processing apparatus 101 and the mobile terminal 103.

  The signature superimposing unit 609 is a functional unit for superimposing the signature received by the portable terminal 103 on the document data or screen projected by the function of the projection control unit 603. The signature superimposing unit 609 may superimpose document data or a screen layer and a signature layer, or may synthesize a signature on the document data or screen.

  The portable terminal 103 includes a storage unit 611, a display control unit 612, a communication control unit 613, and a signature input reception unit 614 as functional units.

  The storage unit 611 is a functional unit for storing information constituting various screens to be described later. Add, update, and delete information as necessary. The storage unit 611 also has a function of storing information received by the communication control unit 613 described later.

  The display control unit 612 is a functional unit for controlling display on the touch panel 510 of the mobile terminal 103. When a display instruction such as a screen is issued from the display control unit 612, the display instruction is transmitted to the touch panel 510 via the output controller 506. When touch panel 510 receives this instruction, touch panel 510 displays a screen or the like according to the instruction.

  The communication control unit 613 is a functional unit for controlling communication between the mobile terminal 103 and the information processing apparatus 101. The communication control unit 613 controls transmission / reception of information between the mobile terminal 103 and the information processing apparatus 101.

  The signature input receiving unit 614 is a functional unit for receiving a signature input via the touch panel 510. A touch operation on the touch panel 510 is received, and the locus of the touch operation is acquired as vector format or raster format data.

  Next, the flow of processing for accepting an operation on the menu screen 800 will be described using the flowchart shown in FIG.

  First, in step S701, the CPU 401 of the information processing apparatus 101 activates a projection application stored in the external memory 411 or the like according to an instruction from the teller 301. The projection application is an application for executing processes in steps S702 to S706 described later. By operating the projection application, the projector 423 can project various screens onto the projection plane 304.

  In step S <b> 702, the CPU 401 of the information processing apparatus 101 displays a menu screen 800 as shown in FIG. 8 on the display 410 by the function of the display control unit 602. The menu screen 800 includes a document entry button 801 and an end button 802. A document entry button 801 is a button for inputting a signature for document data. An end button 802 is a button for ending the projection application.

  In step S <b> 703, the CPU 401 of the information processing apparatus 101 determines whether pressing of the document entry button 801 is detected by the function of the operation detection unit 607. That is, it is determined whether or not pressing of the document entry button 801 is accepted on the display 410 via the input device 409. If it is determined that pressing of the document entry button 801 is detected, the process proceeds to step S704. If it is determined that pressing of the document entry button 801 has not been detected or that pressing of a button other than the document entry button 801 has been detected, the process proceeds to step S705.

  In step S704, the CPU 401 of the information processing apparatus 101 performs document entry processing. Details of the document entry process are shown in FIG. When the document entry process is completed, the process returns to step S703.

  In step S <b> 705, the CPU 401 of the information processing apparatus 101 determines whether pressing of the end button 802 is detected by the function of the operation detection unit 607. That is, it is determined whether or not the end button 802 is pressed on the display 410 via the input device 409. If it is determined that pressing of the end button 802 is detected, the process proceeds to step S706. If it is determined that pressing of the end button 802 is not detected, the process returns to step S703.

  In step S706, the CPU 401 of the information processing apparatus 101 ends the projection application started in step S701.

  Next, the flow of document entry processing will be described using the flowchart shown in FIG. Note that arrows represented by broken lines in FIG. 9 indicate data transmission / reception between the information processing apparatus 101 and the mobile terminal 103.

  In step S901, the CPU 401 of the information processing apparatus 101 causes the projector 423 to project a document entry screen 1000 as shown in FIG. At this time, the same screen may be displayed on the display 410 using the mirroring function, or a manual for operating the screen and points to be explained are displayed on the display 410 using the multi-monitor function. Also good. The same applies to the case where the screen is projected from the projector 423 onto the projection plane 304.

  The document entry screen 1000 includes a document data display area 1001 for displaying document data and a completion button 1003. In the document data display area 1001, document data selected by the user on a document data selection screen (not shown) is displayed. The selected document data is acquired using information stored in the document data storage location 1102 of the document data management table 1100 shown in FIG. In the document data display area 1001, the document data can be enlarged and reduced.

  The document data management table 1100 is a data table including a document ID 1101, a document data storage location 1102, and a signature column position 1103, and is stored in the external memory 411 of the information processing apparatus 101. The document ID 1101 indicates identification information uniquely assigned for each document data. A document data storage location 1102 indicates a file path in which document data is stored. The signature field position 1103 (position information) indicates the upper left coordinates and lower right coordinates of the signature field included in the document data. It should be noted that the center coordinates of the signature field and the vertical and horizontal widths from the center coordinates may be stored. If the document data does not have a signature field, “NULL” is stored. The signature field indicated by the signature field position 1103 is, for example, the signature field area 1002 in FIG. The coordinates indicated by the signature field position 1103 are coordinates in the coordinate system of the screen projected on the projection plane 304.

  In step S <b> 902, the CPU 401 of the information processing apparatus 101 determines whether the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected, by the function of the sensor control unit 605. More specifically, the mobile terminal 103 on the projection plane 304 is detected using information acquired from the infrared sensor 422 connected to the information processing apparatus 101 and the size 1112 of the mobile terminal management table 1110 shown in FIG. To do. The size 1112 stores the height, width, and thickness of the mobile terminal 103, and the mobile terminal 103 can be detected by comparing these information with information acquired from the infrared sensor 422.

  The mobile terminal management table 1110 is a data table including a terminal ID 1111, a size 1112, and a mobile terminal position 1113, and is stored in the external memory 411 of the information processing apparatus 101. The terminal ID 1111 indicates identification information that is uniquely assigned to each mobile terminal 103. The size 1112 indicates the height, width, and thickness of the mobile terminal 103. The mobile terminal position 1113 indicates the upper left coordinates and the lower right coordinates of the mobile terminal 103 on the screen projected on the projection plane 304. In addition, you may store the center coordinate of the portable terminal 103, and the width | variety of the up-down and left-right from a center coordinate. The portable terminal position 1113 is coordinates converted into a coordinate system of a screen projected on the projection plane 304 using information acquired from the infrared sensor 422. The mobile terminal position 1113 is continuously updated while the mobile terminal 103 is detected on the projection plane 304.

  Other methods for detecting the mobile terminal 103 are as follows. Information indicating the mobile terminal 103 is displayed on the touch panel 510 of the mobile terminal 103, and the digital camera 421 captures the projection plane 304. If this information is included in the image data generated by shooting, it may be determined that the portable terminal 103 is detected by the projection plane 304. The method for detecting the mobile terminal 103 on the projection plane 304 is not limited to this. If it is determined that the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected, the process proceeds to step S903. If it is not determined that the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected, the process proceeds to step S915.

  In step S <b> 903, the CPU 401 of the information processing apparatus 101 determines whether the detected mobile terminal 103 is present (arranged) in the signature field area 1002 by the function of the sensor control unit 605. More specifically, the signature field position 1103 of the document data projected on the projection plane 304 is compared with the detected mobile terminal position 1113 of the mobile terminal 103, and the detected mobile terminal 103 exists in the signature field area 1002. It is determined whether or not to do. FIG. 12 is an example when the detected mobile terminal 103 exists in the signature field area 1002. As shown in FIG. 12, it is desirable that the signature field area 1002 exists in a position that fits in the area of the touch panel 510 of the mobile terminal 103, but is not limited thereto. If it is determined that the detected mobile terminal 103 exists in the signature field area 1002, the process proceeds to step S904. If it is not determined that the detected mobile terminal 103 exists in the signature field area 1002, that is, if it is determined that the detected mobile terminal 103 does not exist in the signature field area 1002, the process proceeds to step S915. In this way, when the mobile terminal 103 is present in the signature field area 1002, whether the signature is directly signed on the signature field by accepting the signature via the touch panel 510 of the mobile terminal 103? It is possible to give such a feeling to the user.

  In step S904, the CPU 401 of the information processing apparatus 101 transmits an instruction to accept the input of the signature to the detected mobile terminal 103 by the function of the communication control unit 608 (instruction means). When the process of step S904 is completed, the process proceeds to step S908. On the other hand, in step S <b> 905, the CPU 501 of the portable terminal 103 receives the signature input acceptance instruction transmitted from the information processing apparatus 101 by the function of the communication control unit 613.

  In step S906, the CPU 501 of the portable terminal 103 displays a signature input screen 1300 as shown in FIG. 13 by the function of the display control unit 612 in accordance with the instruction received in step S905. Then, the signature input from the user is received by the function of the signature input receiving unit 614 (touch operation receiving means). On the signature input screen 1300, an area of the document entry screen 1000 corresponding to the position where the portable terminal 103 is placed is displayed, and a signature can be input by performing a touch operation via the touch panel 510. In order to display the area of the document entry screen 1000 corresponding to the position where the portable terminal 103 is placed, the image data captured from the projected document entry screen 1000 and the portable terminal position 1113 are displayed from the information processing apparatus 101. It transmits to the portable terminal 103 and displays using these.

  Since the area of the document entry screen 1000 corresponding to the position where the mobile terminal 103 is placed is displayed on the signature input screen 1300, control is performed using the mobile terminal position 1113 so that this area is not projected from the projector 423. Also good. By doing so, the readability of the screen displayed on the touch panel 510 of the mobile terminal 103 is improved. Further, on the signature input screen 1300, an area on the document entry screen 1000 corresponding to the position where the mobile terminal 103 is placed may not be displayed, but a screen on which a signature can be input may be displayed.

  In step S <b> 907, the CPU 501 of the mobile terminal 103 generates signature information indicating the trace traced by the touch operation by the function of the signature input reception unit 614, acquires it, and temporarily stores it in the RAM 503. The signature information is data in a vector format or a raster format (also referred to as a bitmap format) generated by the signature input reception unit 614, and information obtained from a trace traced by a touch operation, that is, an input received from a user. It is a signature. The signature information is not limited to the vector format or the raster format, and may be data that is converted into text using a known character recognition technique with respect to the locus obtained by the touch operation. After the signature information is generated, the process waits for an instruction from the information processing apparatus 101.

  In step S <b> 908, the CPU 401 of the information processing apparatus 101 determines whether the mobile terminal 103 detected in step S <b> 902 has moved from the signature field area 1002 by the function of the sensor control unit 605. More specifically, the signature field position 1103 of the document data projected on the projection plane 304 is compared with the detected mobile terminal position 1113 of the mobile terminal 103, and the detected mobile terminal 103 moves from the signature field area 1002. Determine whether or not. If it is determined that the mobile terminal 103 has moved from the signature field area 1002, the process proceeds to step S909. If it is not determined that the mobile terminal 103 has moved from the signature field area 1002, that is, if it is determined that the mobile terminal 103 has not moved from the signature field area 1002, the mobile terminal 103 waits until it moves. The fact that the mobile terminal 103 has moved from the signature field area 1002 means that the signature on the mobile terminal 103 has been completed. That is, the information processing apparatus 101 operates to acquire signature information from the mobile terminal 103 by detecting the movement of the mobile terminal 103 from the signature field area 1002. It should be noted that the determination is preferably made in consideration of the fact that the mobile terminal 103 moves somewhat with the input of the signature in the mobile terminal 103.

  In step S909, the CPU 401 of the information processing apparatus 101 instructs the portable terminal 103 on the projection plane 304 to acquire the signature information generated in step S907 by the function of the communication control unit 608. When step S909 is completed, the process waits until signature information is sent from the portable terminal 103. In step S <b> 910, the CPU 501 of the mobile terminal 103 receives the signature information acquisition instruction transmitted from the information processing apparatus 101 by the function of the communication control unit 613.

  In step S911, the CPU 501 of the mobile terminal 103 acquires the signature information stored in the RAM 503 in step S907 according to the instruction received in step S910 by the function of the signature input reception unit 614. In step S912, the CPU 501 of the mobile terminal 103 transmits the signature information acquired in step S911 to the information processing apparatus 101 by the function of the communication control unit 613.

  In step S <b> 913, the CPU 401 of the information processing apparatus 101 receives signature information transmitted from the mobile terminal 103 by the function of the communication control unit 608. That is, the information obtained from the touch operation locus is obtained from the portable terminal 103 (trajectory acquisition means).

  In step S914, the CPU 401 of the information processing apparatus 101 uses the function of the signature superimposing unit 609 to superimpose the signature information received in step S913 on the signature field in the signature field area 1002 of the document data (superimposing unit). Then, the superimposed document data is projected on the projector 423 by the function of the projection control unit 603 and displayed on the projection plane 304. If necessary, the signature information is enlarged or reduced so as to fit in the signature field. FIG. 14 is a diagram illustrating a state in which the signature information is superimposed on the signature field. When the mobile terminal 103 moves from the signature field area 1002, the signature information is transmitted to the information processing apparatus 101, and is displayed so as to be included in the signature field of the signature field area 1002, as shown in FIG.

  In step S <b> 915, the CPU 401 of the information processing apparatus 101 determines whether or not pressing of the completion button 1003 included in the document entry screen 1000 is detected by the function of the operation detection unit 607. That is, it is determined whether or not a touch operation on the completion button 1003 is detected on the document entry screen 1000 projected onto the projection plane 304. If it is determined that pressing of the completion button 1003 is detected, the process proceeds to step S916. If it is determined that pressing of the completion button 1003 has not been detected, the process returns to step S902.

  In step S <b> 916, the CPU 401 of the information processing apparatus 101 stores the document data on which the signature information is superimposed in the external memory 411 using the function of the storage unit 601. When saving, the signature information is combined with the signature column of the displayed document data by the function of the signature superimposing unit 609 to obtain one document data. Further, the function of the storage unit 601 stores necessary information for each item of the signed document management table 1120 shown in FIG.

  The signed document management table 1120 is a data table including a signed document ID 1121, an original document ID 1122, a customer ID 1123, and a signed document storage location 1124, and is stored in the external memory 411 of the information processing apparatus 101. . The signed document ID 1121 indicates identification information uniquely assigned to each document data stored in step S916. The original document ID 1122 indicates the document ID 1101 of the document data to be combined in step S916. The customer ID 1123 indicates identification information uniquely assigned to each customer. The signed document storage location 1124 indicates the file path of the document data stored in step S916. When saving in step S916, a new record is added to the signed document management table 1120, and each piece of information is stored. When step S916 is completed, the document entry process is terminated.

  As described above, according to the present embodiment, it is possible to superimpose the trajectory of the touch operation on the projection data projected by the projection device using the mobile terminal including the touch panel.

  Next, a second embodiment will be described. In the first embodiment described above, the mechanism for superimposing information obtained from the locus on the document data by accepting a touch operation in the portable terminal 103 with the portable terminal 103 placed in the signature section has been described. In the second embodiment, the mobile terminal 103 receives a touch operation before the mobile terminal 103 is placed in the signature field, and the mobile terminal 103 is placed in the signature field, so that information obtained from the trajectory is superimposed on the document data. The mechanism to do is explained.

  Since the second embodiment is a modification of the flowchart of FIG. 9 in the first embodiment described above, other points (system configuration, hardware configuration, functional configuration, flowchart, data table, screen example, etc.) Is the same as in the first embodiment. Therefore, the description of the same points as in the first embodiment will be omitted, and the same reference numerals used in the second embodiment will be used for the reference numerals described in the first embodiment.

  The document entry processing in the second embodiment will be described using the flowchart shown in FIG. Note that arrows represented by broken lines in FIG. 15 indicate data transmission / reception between the information processing apparatus 101 and the portable terminal 103.

  In step S1501, the CPU 401 of the information processing apparatus 101 causes the projector 423 to project a document entry screen 1000 as shown in FIG.

  In step S <b> 1502, the CPU 401 of the information processing apparatus 101 transmits an instruction to accept a signature input to the mobile terminal 103 by the function of the communication control unit 608. When the process of step S1502 is completed, the process proceeds to step S1506. On the other hand, in step S <b> 1503, the CPU 501 of the portable terminal 103 receives a signature input acceptance instruction transmitted from the information processing apparatus 101 by the function of the communication control unit 613.

  In step S1504, the CPU 501 of the portable terminal 103 displays a signature input screen 1300 as shown in FIG. 13 by the function of the display control unit 612 in accordance with the instruction received in step S1503. Then, the signature input from the user is received by the function of the signature input receiving unit 614.

  In step S <b> 1505, the CPU 501 of the portable terminal 103 uses the function of the signature input reception unit 614 to generate and obtain signature information indicating a trace traced by a touch operation, and temporarily stores the signature information in the RAM 503. That is, the user places the mobile terminal 103 at hand and performs a touch operation on the touch panel 510 to sign. The signature information is data in a vector format or a raster format (also referred to as a bitmap format) generated by the signature input reception unit 614, and information obtained from a trace traced by a touch operation, that is, an input received from a user. It is a signature. The signature information is not limited to the vector format or the raster format, and may be data that is converted into text using a known character recognition technique with respect to the locus obtained by the touch operation. After the signature information is generated, the process waits for an instruction from the information processing apparatus 101.

  In step S <b> 1506, the CPU 401 of the information processing apparatus 101 determines whether the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected by the function of the sensor control unit 605. That is, the user places the signed portable terminal 103 on the projection plane 304 to cause the information processing apparatus 101 to detect it. More specifically, the mobile terminal 103 on the projection plane 304 is detected using information acquired from the infrared sensor 422 connected to the information processing apparatus 101 and the size 1112 of the mobile terminal management table 1110 shown in FIG. To do. The size 1112 stores the height, width, and thickness of the mobile terminal 103, and the mobile terminal 103 can be detected by comparing these information with information acquired from the infrared sensor 422.

  Other methods for detecting the mobile terminal 103 are as follows. Information indicating the mobile terminal 103 is displayed on the touch panel 510 of the mobile terminal 103, and the digital camera 421 captures the projection plane 304. If this information is included in the image data generated by shooting, it may be determined that the portable terminal 103 is detected by the projection plane 304. The method for detecting the mobile terminal 103 on the projection plane 304 is not limited to this. If it is determined that the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected, the process proceeds to step S1507. If it is not determined that the portable terminal 103 is detected on the projection surface 304 on which the document entry screen 1000 is projected, the process proceeds to step S1516.

  In step S <b> 1507, the CPU 401 of the information processing apparatus 101 determines whether the detected mobile terminal 103 exists in the signature field area 1002 by the function of the sensor control unit 605. More specifically, the signature field position 1103 of the document data projected on the projection plane 304 is compared with the detected mobile terminal position 1113 of the mobile terminal 103, and the detected mobile terminal 103 exists in the signature field area 1002. It is determined whether or not to do. If it is determined that the detected mobile terminal 103 exists in the signature field area 1002, the process proceeds to step S1508. If it is not determined that the detected mobile terminal 103 exists in the signature field area 1002, that is, if it is determined that the detected mobile terminal 103 does not exist in the signature field area 1002, the process proceeds to step S1516.

  In step S1508, the CPU 401 of the information processing apparatus 101 instructs the portable terminal 103 on the projection plane 304 to acquire the signature information generated in step S1505 by the function of the communication control unit 608. When step S1508 is completed, the process waits until signature information is sent from the portable terminal 103. In step S <b> 1509, the CPU 501 of the portable terminal 103 receives the signature information acquisition instruction transmitted from the information processing apparatus 101 by the function of the communication control unit 613.

  In step S1510, the CPU 501 of the portable terminal 103 acquires the signature information stored in the RAM 503 in step S1505 in accordance with the instruction received in step S1509 by the function of the signature input reception unit 614. In step S <b> 1511, the CPU 501 of the mobile terminal 103 transmits the signature information acquired in step S <b> 1510 to the information processing apparatus 101 by the function of the communication control unit 613.

  When the transmission of the signature information to the information processing apparatus 101 is completed, in step S1512, the CPU 501 of the portable terminal 103 deletes the signature information stored in the RAM 503 in step S1505 by the function of the signature input reception unit 614 (trajectory deletion unit). . Deletion may be either logical deletion or physical deletion. In this manner, by deleting (invalidating) the signature information once transmitted to the information processing apparatus 101, it is possible to prevent the signature information from being used in other signature fields.

  In step S <b> 1513, the CPU 401 of the information processing apparatus 101 receives signature information transmitted from the mobile terminal 103 by the function of the communication control unit 608.

  In step S <b> 1514, the CPU 401 of the information processing apparatus 101 determines whether the portable terminal 103 detected in step S <b> 1506 has moved from the signature field area 1002 by the function of the sensor control unit 605. More specifically, the signature field position 1103 of the document data projected on the projection plane 304 is compared with the detected mobile terminal position 1113 of the mobile terminal 103, and the detected mobile terminal 103 moves from the signature field area 1002. Determine whether or not. If it is determined that the mobile terminal 103 has moved from the signature field area 1002, the process proceeds to step S1515. If it is not determined that the mobile terminal 103 has moved from the signature field area 1002, that is, if it is determined that the mobile terminal 103 has not moved from the signature field area 1002, the mobile terminal 103 waits until it moves. If the received signature information is superimposed and displayed before the portable terminal 103 moves from the signature field area 1002, the signature information is further superimposed on the portable terminal 103, making it difficult to see. Therefore, until the portable terminal 103 moves from the signature field area 1002, it is controlled not to be superimposed and displayed.

  In step S1515, the CPU 401 of the information processing apparatus 101 uses the function of the signature superimposing unit 609 to superimpose the signature information received in step S1513 on the signature column in the signature column area 1002 of the document data. Then, the superimposed document data is projected on the projector 423 by the function of the projection control unit 603 and displayed on the projection plane 304. If necessary, the signature information is enlarged or reduced so as to fit in the signature field.

  In step S <b> 1516, the CPU 401 of the information processing apparatus 101 determines whether pressing of the completion button 1003 included in the document entry screen 1000 is detected by the function of the operation detection unit 607. That is, it is determined whether or not a touch operation on the completion button 1003 is detected on the document entry screen 1000 projected onto the projection plane 304. If it is determined that pressing of the completion button 1003 has been detected, the process proceeds to step S1517. If it is determined that pressing of the completion button 1003 has not been detected, the process returns to step S1506.

  In step S1517, the CPU 401 of the information processing apparatus 101 stores the document data on which the signature information is superimposed in the external memory 411 using the function of the storage unit 601 (storing unit). When saving, the signature information is combined with the signature column of the displayed document data by the function of the signature superimposing unit 609 to obtain one document data. Further, the function of the storage unit 601 stores necessary information for each item of the signed document management table 1120 shown in FIG.

  FIG. 16 is a diagram showing an outline of the second embodiment. First, when the information processing apparatus 101 causes the projector 423 to project document data, the portable terminal 103 can accept a signature input. Then, as shown in FIG. 16A, the user inputs a signature to the mobile terminal 103. In response to this, the portable terminal 103 draws a locus, and temporarily stores information obtained from the locus, that is, signature information in the RAM 503.

  When the signature is completed, the user places the portable terminal 103 in the signature field area 1002 as shown in FIG. When the information processing apparatus 101 determines that the mobile terminal 103 is placed in the signature field area 1002, the information processing apparatus 101 acquires signature information from the mobile terminal 103. When the signature information is acquired, the signature information is deleted from the storage area of the mobile terminal 103. When the user moves the mobile terminal 103 from the signature field area 1002, the information processing apparatus 101 detects this and displays the acquired signature information in the signature field area 1002 in a superimposed manner. The result is shown in FIG.

  In this way, even when the mobile terminal 103 is not placed in the signature field area 1002 and the signature is not performed, the mobile terminal 103 is signed by the user and then the mobile terminal 103 is placed in the signature field area 1002. The signature can be easily performed using the portable terminal 103.

  As described above, according to the present embodiment, it is possible to superimpose the trajectory of the touch operation on the projection data projected by the projection device using the mobile terminal including the touch panel.

  The present invention can be implemented as a system, apparatus, method, program, storage medium, or the like, and can be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  Note that the present invention includes a software program that implements the functions of the above-described embodiments directly or remotely from a system or apparatus. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Therefore, in order to realize (executable) the functional processing of the present invention by a computer, the program code itself installed in the computer also realizes the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  Note that the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101 Information Processing Device 102 Projection Imaging Device 103 Portable Terminal 401 CPU
402 RAM
403 ROM
404 System bus 405 General-purpose input / output controller 406 Video controller 407 Memory controller 408 Communication I / F controller 409 Input device 410 Display 411 External memory 421 Digital camera 422 Gesture sensor 423 Projector

Claims (11)

An information processing system including a projection device that projects projection data, a portable terminal that acquires a trajectory of a touch operation on a touch panel, and an information processing device,
The information processing apparatus includes:
Storage means for storing position information indicating a position in the projection data;
Trajectory acquisition means for acquiring information obtained from the trajectory of the touch operation from the mobile terminal arranged at the position indicated by the positional information stored in the storage means and in the projection data projected by the projection device When,
An information processing system comprising: a superimposing unit that superimposes information obtained from the trajectory acquired by the trajectory acquiring unit on a position in the projection data indicated by the position information. The information processing system further includes a sensor device capable of detecting the position of the mobile terminal in the projection data projected by the projection device,
The information according to claim 1, wherein the trajectory acquisition unit acquires information obtained from the trajectory of the touch operation from the mobile terminal detected at a position indicated by position information stored in the storage unit. Processing system. The portable terminal is
The information according to claim 1, further comprising: a trajectory deleting unit that deletes the information from the mobile terminal when information obtained from the trajectory of the touch operation is acquired by the trajectory acquiring unit. Processing system. In the projection data indicated by the position information, the superimposing means displays information obtained from the trajectory acquired by the trajectory acquisition means when the mobile terminal moves from the position indicated by the positional information stored in the storage means. The information processing system according to any one of claims 1 to 3, wherein the information processing system is superimposed on a position. The information processing apparatus includes:
An instruction means for instructing the portable terminal arranged at the position indicated by the position information stored in the storage means to accept the touch operation;
The portable terminal is
The information processing system according to any one of claims 1 to 4, further comprising: a touch operation accepting unit that accepts the touch operation when an instruction is accepted by the instructing unit. The information processing apparatus includes:
The information processing system according to claim 1, further comprising: a storage unit that stores projection data on which the information indicating the locus is superimposed by the superimposing unit. A control method for an information processing system, comprising: a projection device that projects projection data; a portable terminal that acquires a trajectory of a touch operation on the touch panel; and an information processing device that includes storage means for storing position information indicating a position in the projection data. Because
The trajectory of the touch operation from the portable terminal arranged at the position indicated by the position information stored in the storage means, in the trajectory acquisition means of the information processing apparatus is the position in the projection data projected by the projection apparatus. A trajectory acquisition step for acquiring information obtained from
An information processing system comprising: a superimposing step, wherein the superimposing unit of the information processing apparatus superimposes information obtained from the trajectory acquired in the trajectory acquiring step on a position in the projection data indicated by the position information. Control method. A control method for an information processing system, comprising: a projection device that projects projection data; a portable terminal that acquires a trajectory of a touch operation on the touch panel; and an information processing device that includes storage means for storing position information indicating a position in the projection data. A program that can execute
The information processing apparatus;
Trajectory acquisition means for acquiring information obtained from the trajectory of the touch operation from the mobile terminal arranged at the position indicated by the positional information stored in the storage means and in the projection data projected by the projection device When,
A program that causes information obtained from a trajectory acquired by the trajectory acquisition unit to function as a superimposing unit that superimposes the information on the position in the projection data indicated by the position information. An information processing apparatus capable of communicating with a projection apparatus that projects projection data and a mobile terminal that acquires a trajectory of a touch operation on a touch panel,
Storage means for storing position information indicating a position in the projection data;
Trajectory acquisition means for acquiring information obtained from the trajectory of the touch operation from the mobile terminal arranged at the position indicated by the positional information stored in the storage means and in the projection data projected by the projection device When,
An information processing apparatus comprising: a superimposing unit that superimposes information obtained from the trajectory acquired by the trajectory acquiring unit on a position in the projection data indicated by the position information. A control method for an information processing apparatus that is communicable with a projection apparatus that projects projection data and a portable terminal that obtains a trajectory of a touch operation on a touch panel and includes a storage unit that stores position information indicating a position in the projection data. And
The trajectory of the touch operation from the portable terminal arranged at the position indicated by the position information stored in the storage means, in the trajectory acquisition means of the information processing apparatus is the position in the projection data projected by the projection apparatus. A trajectory acquisition step for acquiring information obtained from
An information processing apparatus comprising: a superimposing step in which the superimposing unit of the information processing apparatus superimposes information obtained from the trajectory acquired in the trajectory acquiring step on a position in the projection data indicated by the position information. Control method. A control method of an information processing apparatus that is communicable with a projection apparatus that projects projection data and a portable terminal that acquires a trajectory of a touch operation on the touch panel and that includes storage means for storing position information indicating a position in the projection data is executed. A possible program,
The information processing apparatus;
Trajectory acquisition means for acquiring information obtained from the trajectory of the touch operation from the mobile terminal arranged at the position indicated by the positional information stored in the storage means and in the projection data projected by the projection device When,
A program that causes information obtained from a trajectory acquired by the trajectory acquisition unit to function as a superimposing unit that superimposes the information on the position in the projection data indicated by the position information.

Images