JP2016119019A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operation accuracy, while maintaining intuitive operability in touch operation, even when a part of a finger or a hand of a user hides a display area near an operation point on a screen.SOLUTION: A camera installed in a head-mounted display captures an area of a display section of a mobile terminal, to obtain an image of the area. An image displayed in the display section of the mobile terminal is received. A difference between the received image and the image captured is calculated, to display an image of the calculated difference, on the head-mounted display.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In mobile devices typified by smartphones and tablet terminals, a user accepts an operation by touching a screen (touch panel, touch screen) on which buttons or the like are displayed so that the user can operate intuitively.

  Thus, when the user operates a touch screen such as a mobile device, a part of the user's finger or hand blocks the display area around the operation point on the screen. Therefore, the user cannot visually recognize the actual position of the point to be touched with the finger, and cannot accurately grasp the position.

  Since the user must operate while estimating the touch point on the screen, it is necessary to perform an operation on the screen using senses and intuition. Therefore, the user may not be able to touch a desired point, and an operation error often occurs.

  For example, consider a text editing operation including a mail or a URL of a Web browser, or a drawing operation using a finger. When the user moves the caret (cursor) to a position where he / she wants to input characters, the position may not be specified well for the above reason. The user repeats the touch operation several times or moves the caret to a desired position while using the cursor button, so that the number of operations increases.

  In addition, when the user performs an operation of connecting the lines, the touch operation of drawing the line and canceling the failed operation may be repeated several times.

  As a technique for improving the operation accuracy with respect to such a problem that the number of operations increases, an enlarged display function and a touch pen around the caret can be cited.

  In Patent Literature 1, when a touch panel is touched with a finger in a state where a plurality of selection buttons are displayed on the touch panel, the selection buttons located around the finger contact portion are enlarged and rearranged, It describes the mechanism for displaying the selection button adjacent to it.

JP 2008-77272 A

  The enlarged display function around the caret is a method of displaying an enlarged image in a different window with the display content of the area around the touch point when moving the caret, and the display content of the screen area where the user's finger is supposed to cover the screen Is presented at a position other than the touch point. Thereby, since the user can recognize the operation state with respect to the touch point and its surroundings, it can be said that the operation error and the number of touch operations required for the recovery operation are suppressed.

  However, depending on the window display location, the enlarged display window may be hidden by the user's finger or hand, and the objective may not be achieved successfully. Further, since the user needs to know the touch operation for the enlarged display, it is difficult to say that the operation is intuitive.

  As for the touch pen, since the pen tip is thinner than the fingertip, the visibility of information displayed around the touch point is improved, and the accuracy of the touch operation can be improved.

  However, since a pen is not used when performing a multi-touch operation that is an intuitive operation method unique to the touch operation interface, a situation occurs in which operations are performed when the pen is used and when the pen is not used. In such a case, it is conceivable that the change of the pen frequently occurs, so that the screen operation becomes complicated.

  Therefore, an object of the present invention is to provide a mechanism capable of improving the accuracy of operation while maintaining the intuitive operability of the touch operation.

  An information processing apparatus according to the present invention captures an area of a display unit of a mobile terminal, acquires an image of the area, and receives an image displayed on the display unit of the mobile terminal from the mobile terminal And means for calculating a difference between the image received by the receiving means and the image acquired by the acquiring means, and a display means for displaying an image of the difference calculated by the calculating means. Features.

  In the information processing method of the present invention, the acquisition unit of the information processing device captures an area of the display unit of the mobile terminal and acquires an image of the area, and the reception unit of the information processing apparatus includes the mobile terminal. The receiving step of receiving the image displayed on the display unit from the portable terminal, and the calculation means of the information processing device calculates the difference between the image received by the receiving step and the image acquired by the acquiring step And a display step in which the display means of the information processing apparatus displays a difference image calculated in the calculation step.

  The program of the present invention is a program that can be executed in an information processing device, and the information processing device captures an area of a display unit of a mobile terminal and acquires an image of the area; An image displayed on the display unit is calculated by a receiving unit that receives from the portable terminal, a calculating unit that calculates a difference between the image received by the receiving unit and the image acquired by the acquiring unit, and is calculated by the calculating unit It is made to function as a display means for displaying the difference image.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to improve the precision of operation, maintaining the intuitive operativity of touch operation.

System configuration diagram showing an example of a system configuration of the present invention The block diagram which shows the hardware constitutions of the portable terminal 100 and the information processing apparatus 101 Block diagram showing functions of portable terminal 100 and information processing apparatus 101 Flow chart showing an example of image display processing Flowchart showing an example of display image data transmission processing The flowchart which shows an example of touch point information transmission processing Diagram showing an example of a marker The figure which shows an example of the portable terminal which provided the marker The figure which shows an example of the screen in the portable terminal at the time of touch operation The figure which shows an example of the screen in the portable terminal at the time of touch operation The figure which shows an example of the screen in the portable terminal at the time of touch operation

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram showing an example of a system configuration of the present invention.

  FIG. 1 shows a configuration in which one mobile terminal 100 and one information processing apparatus 101 are communicably connected.

  The mobile terminal 100 is a mobile terminal equipped with an operation system having a user interface function that presents information to a user and receives a user's touch operation as an input. For example, a smartphone or a tablet terminal is a suitable example.

  The information processing apparatus 101 is an apparatus that can display an image superimposed on the user's field of view. In the present invention, the user interface image is displayed in an overlay according to the situation in which the user operates the mobile terminal 100. A so-called head-mounted display is a suitable example.

  In the system according to the present embodiment, a physical keyboard is not connected to one mobile terminal 100 and one information processing apparatus 101 having the above-described configuration, or even if a physical keyboard is connected, the system is not used. May be.

  FIG. 2 is a block diagram illustrating a hardware configuration of the mobile terminal 100 and the information processing apparatus 101 illustrated in FIG.

  In FIG. 2, reference numeral 201 denotes a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 213 has a BIOS (Basic Input / Output System) or an operating system program (hereinafter referred to as an OS), which is a control program of the CPU 201, and functions necessary for realizing functions executed by each mobile terminal. Various programs to be stored are stored.

A RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for execution of processing from the ROM 203 or the external memory (flash memory (ROM)) 213 to the RAM 202 and executing the loaded program.

  An input controller 205 controls input from a keyboard (KB) 209, the touch panel 210, the camera 211, or a sensor such as an acceleration sensor (not shown).

  The keyboard 209 includes a device for external connection. The touch panel 210 includes both a detection-type device that can sense a plurality of touch points on the screen and a detection-type device that can sense the position of a finger that is slightly away from the screen.

  There is a mutual capacitive touch panel as a technology that can detect a plurality of touch points on the screen. In addition, there is a self-capacitance type touch panel as a technique that can sense the position of a finger that is slightly away from the screen.

  Alternatively, as a method for detecting the position of a finger that is slightly away from the screen, it is possible to use an analysis of an image on which the screen and the finger are reflected, so the camera 211 may be used as the image input device.

  The camera 211 is also a means for acquiring position information necessary for image display processing in the information processing apparatus 101. In order to determine the attitude of the mobile terminal by reading a marker added to the mobile terminal from the camera image. Use.

  A video controller 206 controls display on a display device such as the display 212. The touch panel 210 is overlaid on the display 212 so as to accept a user's touch operation. The display shown in FIG. 2 is not limited to a liquid crystal display, but may be another display such as an organic EL.

  A memory controller 207 controls access to the external memory 213 that stores a boot program, various applications, font data, user files, edit files, various data, and the like.

  A communication I / F controller 208 connects and communicates with an external device via a network (for example, a communication network such as 3G (not shown) or a wireless LAN), and executes communication control processing in the network. For example, communication using TCP / IP is possible.

  Note that the CPU 201 enables display on the display 212 by executing, for example, an outline font rasterization process on a display information area in the RAM 202. In addition, the CPU 201 enables a user instruction through a user interface (not shown) on the display 212.

  Various programs to be described later for realizing the present invention are recorded in the ROM 203 or the external memory 213, and are executed by the CPU 201 by being loaded into the RAM 202 as necessary. Furthermore, definition files and various information tables used when executing the program are also stored in the ROM 203 or the external memory 213, and detailed descriptions thereof will be described later.

  Next, a functional block diagram of the mobile terminal 100 as the information processing apparatus of the present invention will be described with reference to FIG.

  In addition, the detailed control which each functional block processes is demonstrated with the flowchart mentioned later.

  In FIG. 3, the display display image acquisition unit 301 stores the image data displayed on the display 212 in the ROM 203 or the external memory 213.

  When receiving a request from the information processing apparatus 101, the display display image transmission unit 302 transmits the image data acquired by the display display image acquisition unit 301 to the information processing apparatus 101.

  The touch information acquisition unit 303 acquires information on whether or not the touch panel 210 has detected a finger and coordinates on the screen of the detected finger (or coordinates on which the finger on the screen is closest).

  When there is a request from the information processing device 101, the touch information transmission unit 304 transmits the touch point information acquired by the touch point information acquisition unit 303 to the information processing device 101.

  Next, a functional block diagram of the information processing apparatus 101 as the information processing apparatus of the present invention will be described with reference to FIG.

In addition, the detailed control which each functional block processes is demonstrated with the flowchart mentioned later.

  In FIG. 3, a camera image acquisition unit 305 stores image data captured by the camera 211 of the information processing apparatus 101 in the ROM 203 or the external memory 213.

  The marker information acquisition unit 306 analyzes image data captured by the camera 211 and determines the posture of the marker added to the mobile terminal 100.

  The display display image receiving unit 307 receives display display image data requested to the mobile terminal 100.

  The touch information receiving unit 308 receives touch point information requested to the mobile terminal 100.

  The image generation unit 309 generates image data to be overlaid from the marker posture acquired by the marker information acquisition unit 306 and the image data acquired by the display display image reception unit 307.

The image display unit 310 renders the image data generated by the image generation unit 309 on the display 212 of the information processing apparatus 101. In the example of FIG. 1, the lens portion of the glasses becomes the image display unit 310, and an image generated by the image generation unit 309 is drawn on the lens portion, so that an image is visually input to the user.
The overall system flow in the present embodiment will be described below.

  When the camera 211 of the information processing apparatus 101 detects a marker attached to the mobile terminal 100, the information processing apparatus requests the mobile terminal 100 to transmit image data. Further, the information processing apparatus 101 obtains the posture information of the mobile terminal 100 and the perspective transformation matrix from the detected marker information.

  When the portable terminal 100 receives a request for image data transmission, the portable terminal 100 transmits display image data to the information processing apparatus 101. The information processing apparatus 101 converts the image data received from the mobile terminal 100 using a perspective transformation matrix, and obtains a difference from the image in the marker area captured by the camera 211. As a result, the difference between the area on the screen covered with the user's finger or hand on the portable terminal 100 (the image displayed on the portable terminal 100 and the image in the marker area photographed by the camera 211 of the information processing apparatus 101) ) Specifies the information processing apparatus 101.

  The information processing apparatus 101 displays the converted display image data so as to overlap the difference area. In addition, when the distance between the screen of the portable terminal 100 and the finger is within a certain range, the information processing apparatus 101 further displays an image indicating a touch point on the screen in an overlapping manner.

  Hereinafter, the UI display process in the system of the present embodiment will be described with reference to FIG.

  The process shown in the flowchart of FIG. 4 is a process in which the CPU 201 of the information processing apparatus 101 reads and executes a predetermined control program.

  In step S401, the camera 211 of the information processing apparatus 101 photographs the mobile terminal 100 with a marker added around the screen. Then, the photographed image is analyzed to extract (acquire) the marker image pattern.

  An example of the marker is shown in FIG. An example of the mobile terminal 100 to which a marker is added is shown in FIG.

  In this embodiment, the marker is added so as to surround the periphery of the screen of the mobile terminal 100 as shown in FIG. 7, but the mode of the marker is not limited to this and recognizes the screen area of the mobile terminal. Any embodiment may be used as long as it can be applied.

  In step S402, it is determined whether the marker image pattern acquired in step S401 matches (matches) a previously registered marker image pattern.

If they match (step S402: YES), the process proceeds to step S403.
If they do not match, the process of this flowchart is terminated.

  In step S403, four vertices of the marker area are acquired from the marker image acquired in step S401, and a projective transformation matrix from the display image area of the portable terminal 100 to the marker area is calculated. However, it is necessary to correct the marker coordinates with respect to the viewpoint of the user wearing the information processing apparatus 101.

In step S <b> 404, the information processing apparatus 101 requests the mobile terminal 100 to transmit display image data. Receiving the display image data transmission request, the portable terminal 100 executes display image data transmission processing. Then, the information processing apparatus 101 receives display image data from the mobile terminal 100.
Details of the display image data transmission processing will be described with reference to FIG.

  The process shown in the flowchart of FIG. 5 is a process in which the CPU 201 of the mobile terminal 100 reads and executes a predetermined control program.

  In step S <b> 501, the CPU 201 of the mobile terminal 100 receives an image data transmission request from the information processing apparatus 101.

  In step S502, the CPU 201 of the mobile terminal 100 acquires image data displayed on the screen (display unit).

In step S503, the CPU 201 of the portable terminal 100 transmits the image data acquired in step S502 to the information processing apparatus 101.
Returning to the description of FIG.

  In step S405, the information processing apparatus 101 multiplies the display image data acquired in step S404 by the projection transformation matrix calculated in S403, and generates projection image data from the display image data to the marker area.

  Then, the information processing apparatus 101 acquires the difference between the image in the marker area in the image captured by the camera 211 and the projected image data. Since this difference image has been converted into the marker coordinate system, it matches the region of the user's finger or part of the hand covering the screen (display unit) of the mobile terminal 100 as viewed from the information processing apparatus 101.

  In step S406, it is determined whether a difference image has been acquired in step S405.

  If the difference image has been acquired (step S406: YES), the process proceeds to step S407.

  If the difference image cannot be acquired (step S406: NO), that is, if the display area is not covered with a part of the user's finger or hand, the process proceeds to step S401.

  In step S407, the information processing apparatus 101 displays the difference image acquired in step S405 on the display 212 of the information processing apparatus 101.

  At this time, when the user of the information processing apparatus 101 views the mobile terminal 100 through the screen, it can be confirmed that the difference image is displayed on the actual video. That is, an image being displayed on the screen of the mobile terminal 100 is displayed for a portion where the user's finger covers the screen of the mobile terminal 100 in the marker area (an example is shown in FIG. 10).

  In step S <b> 408, the information processing apparatus 101 requests the portable terminal 100 to transmit touch point information.

  The mobile terminal 100 that has received the request for touch point information transmission executes touch point information transmission processing (FIG. 6) and transmits the touch point information to the information processing apparatus 101. Then, the information processing apparatus 101 receives the transmitted touch point information.

  Details of the touch point information transmission processing will be described with reference to the flowchart of FIG.

  The process shown in the flowchart of FIG. 6 is a process in which the CPU 201 of the mobile terminal 100 reads and executes a predetermined control program.

  In step S <b> 601, the CPU 201 of the mobile terminal 100 receives a touch point information transmission request from the information processing apparatus 101.

  In step S602, the CPU 201 of the mobile terminal 100 acquires touch point information on the screen. Specifically, the point where the distance between the user's finger or stylus pen and the screen is closest is acquired.

In step S <b> 603, the CPU 201 of the mobile terminal 100 transmits the touch point information acquired in step S <b> 602 to the information processing apparatus 101.
Returning to the description of FIG.

  In step S409, the information processing apparatus 101 refers to the touch point information acquired in step S408, and determines whether the finger and the display are within a certain distance.

  If within a certain distance (step S409: YES), the process proceeds to step S410.

  If it is not within the certain distance (step S409: NO), the process returns to step S401.

  In step S410, the information processing apparatus 101 multiplies the touch point information acquired in step S408 by the projective transformation matrix calculated in step S403, and calculates the converted touch point coordinates in the marker area.

  The information processing apparatus 101 displays the pointer image superimposed on the display 212 with respect to the converted touch point coordinates. At this time, a pointer image indicating a touch point is further displayed on the difference image in step S407 for a portion where the user's finger covers the screen of the portable terminal 100 in the marker area. An example is shown in FIG.

  FIG. 9 is a diagram illustrating a state where a finger, a stylus pen, or the like (901) covers the display unit on the screen of the mobile terminal 100.

  FIG. 10 is a diagram illustrating an example of a screen display on the information processing apparatus 101 in the system according to the present embodiment. When the user performs a touch operation, the finger blocks the screen of the portable terminal 100 and a part of the screen becomes invisible, but the information processing apparatus 101 displays the difference image 1000 on the display 212 for the area. At that time, by displaying the frame line (contour line) of the difference image 1000, the user can recognize the area occupied by the finger or hand operated by himself / herself, so which part of the screen of the mobile terminal 100 is operated. I understand.

  FIG. 11 is a diagram illustrating an example of pointer image display on the information processing apparatus 101 in the system according to the present embodiment. Since the user can recognize the position of the touch point being operated by the pointer image 1100 displayed by the information processing apparatus 101 on the difference image, errors in the touch operation can be suppressed.

  As described above, according to the present invention, when the user operates the touch screen, the user can determine the screen information around the touch point where the view is blocked by a part of the user's finger or hand and the position of the touch point. Since it can be recognized, the operation accuracy is improved.

  For example, the caret (pointer, cursor) can be moved to a desired position by an intuitive touch operation on a character string displayed on the screen.

  Alternatively, it is possible to easily touch a desired link even when links displayed on the Web browser are close to each other. In addition, the accuracy can be improved in a fine operation that is not good at a touch operation using a finger.

  Moreover, the program in this invention is a program which makes a computer perform the process of FIGS. The program in the present invention may be a program for each process in FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  The present invention may be applied to a system constituted by a plurality of devices or an apparatus constituted by a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

100 mobile terminal 101 information processing apparatus

Claims (5)

An acquisition unit that captures an image of the area of the display unit of the mobile terminal and acquires an image of the area;
Receiving means for receiving an image displayed on the display unit of the mobile terminal from the mobile terminal;
Calculating means for calculating a difference between the image received by the receiving means and the image acquired by the acquiring means;
Display means for displaying an image of the difference calculated by the calculation means;
An information processing apparatus comprising: Touch point information acquisition means for acquiring information indicating a position touched by the user in the display unit of the mobile terminal;
The information processing apparatus according to claim 1, wherein the display unit displays an image indicating a touch point on information indicating the position acquired by the touch point information acquisition unit.   The information processing apparatus according to claim 1, wherein the display unit displays the difference image by adding a contour line of the image. An information processing method in an information processing apparatus,
An acquisition step in which the acquisition unit of the information processing apparatus captures an image of the display unit of the mobile terminal and acquires an image of the area;
A receiving step in which the receiving unit of the information processing apparatus receives an image displayed on the display unit of the mobile terminal from the mobile terminal;
A calculating step of calculating a difference between the image received by the receiving step and the image acquired by the acquiring step;
A display step in which the display means of the information processing apparatus displays an image of the difference calculated by the calculation step;
An information processing method comprising: A program executable in the information processing apparatus,
The information processing apparatus;
An acquisition unit that captures an image of the area of the display unit of the mobile terminal and acquires an image of the area;
Receiving means for receiving an image displayed on the display unit of the mobile terminal from the mobile terminal;
Calculating means for calculating a difference between the image received by the receiving means and the image acquired by the acquiring means;
A program that functions as a display unit that displays an image of a difference calculated by the calculation unit.

Images