JP2014219800A - Portable terminal device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a user to intuitively operate a wide range of devices, considering a three-dimensional arrangement.SOLUTION: A portable terminal device comprises: a display unit provided on the front surface of a housing and a touch panel integrated with the display unit; a camera provided on the back surface of the housing and having a field of view in directions of the back surface and the normal of the display unit; display means for displaying an image obtained by the camera on the display unit in real time; positional information calculation means for configuring a virtual three-dimensional space on the basis of the image obtained by the camera, detecting that an object is included in the obtained image, and calculating positional information in the virtual three-dimensional space; intersection detection means for, when touch input is performed on the touch panel, detecting an intersection of a normal vector passing through a touch input position on the touch panel and a recognition area previously set to the object positioned in the virtual three-dimensional space; and operation message transmission means for transmitting an operation message to a control device of the object on the basis of the detected intersection position.

Description

  The present invention relates to a technique for controlling another device by a mobile terminal device.

  In recent years, monitor screens of televisions and the like have been expanding, and accordingly, the viewing position has also moved away from the monitor screen.

  In addition, the input operation for a mobile terminal device such as a smartphone is generally a touch input to a touch panel provided in the mobile terminal device, and it seems that the object displayed on the screen of the mobile terminal device is directly touched. Intuitive operation is accepted.

  To control the display of the monitor screen, it must be remote operation. However, a video camera is installed in the portable terminal device at hand, the contour of the monitor screen is extracted from the captured image, and displayed on the touch panel. A pointing device that can directly indicate coordinates on a screen has been proposed (see, for example, Patent Document 1).

JP-A-7-104928

  Since the pointing device disclosed in Patent Document 1 extracts the contour of the monitor screen from the image, the three-dimensional information (position information in the space) of the monitor screen is not acquired, and the monitor screen becomes a video camera. On the other hand, when not facing directly, the input to the touch panel may not be accurately converted to coordinates on the monitor screen.

  For example, if the extracted contour is a trapezoid, it cannot be distinguished whether the monitor screen is trapezoidal and directly facing the video camera, or whether the monitor screen is rectangular and tilted to the video camera.

  Further, the pointing device disclosed in Patent Document 1 can be applied only to a monitor screen and cannot be used to control other various devices.

  The present invention has been proposed in view of the above-described conventional problems, and an object of the present invention is to enable intuitive operation with respect to a wide range of devices in consideration of a three-dimensional arrangement. is there.

  In order to solve the above problems, in the present invention, a display unit provided on the front surface of the housing, a touch panel integrated with the display unit, a back surface of the housing, A virtual three-dimensional space is constructed based on a camera having a field of view in the rear and normal directions, display means for displaying a captured image of the camera on the display unit in real time, and the captured image of the camera, and the captured image Position information calculating means for detecting the presence of an object in the virtual three-dimensional space and calculating the position information in the virtual three-dimensional space, and when there is a touch input on the touch panel, the touch input position on the touch panel is An intersection detection means for detecting an intersection between a passing normal vector and a recognition area set in advance on the object positioned in the virtual three-dimensional space, and based on the detected position of the intersection, So that and an operating message transmitting means for transmitting the operation message to the control unit of the object.

  In the present invention, a wide range of devices can be intuitively operated in consideration of a three-dimensional arrangement.

It is a figure which shows the structural example of the system concerning one Embodiment of this invention. It is a figure which shows the example which used the big screen monitor as an operation target. It is a figure which shows the example of the marker which sets the recognition area | region of the operation target object. It is a figure which shows the example using a lighting fixture as an operation target. It is a figure which shows the structural example of a portable terminal device. It is a figure which shows the structural example of an operation target object control apparatus. It is a flowchart which shows the process example of embodiment. It is a figure which shows the concept of the operation target position calculation in real space. It is a figure which shows the example of the superimposition of the image on the screen / touch panel of a portable terminal device.

  Hereinafter, preferred embodiments of the present invention will be described.

<Configuration>
FIG. 1 is a diagram showing a configuration example of a system according to an embodiment of the present invention.

  In FIG. 1, a mobile terminal device 1 is an information processing device such as a tablet or a smartphone. A screen / touch panel 12 is provided on the front surface, and a camera 11 is provided on the back surface. The camera 11 is arranged so as to have a field of view in the normal direction of the back direction of the screen / touch panel 12.

  In addition, one or a plurality of operation objects 2 to be operated are provided by the mobile terminal device 1, and the operation object control device 3 that controls the operation object 2 is provided in the operation object 2. The portable terminal device 1 and the operation target control device 3 communicate with each other by WiFi, Bluetooth (registered trademark), infrared rays, or the like. In addition, although the operation target object 2 and the operation target object control apparatus 3 were shown as a different body, the operation target object 2 and the operation object control apparatus 3 may be comprised integrally.

  The operation object 2 is imaged by the camera 11 of the mobile terminal device 1 and the user touches and operates (tap, drag and drop, flick, etc.) on the screen / touch panel 12 on which the captured content is displayed in real time. Thus, the operation target object 2 can be operated via the operation target object control device 3.

  FIG. 2 is a diagram showing an example in which a large screen monitor (image display device) is used as the operation object 2. In this case, the buttons and characters displayed on the large screen monitor can be operated by touching the screen of the mobile terminal device 1.

  FIG. 3 is a diagram illustrating an example of the marker 21 that sets the recognition area of the operation target object 2. FIG. 3A shows a rectangular area formed by four markers 21 set as a recognition area by sticking the markers 21 to the four corners of the screen with a card or a sticker. The marker 21 describes figures, symbols, and the like in a predetermined format. By analyzing this from an image captured by the camera 11 of the mobile terminal device 1, the three-dimensional position and marker of each marker 21 are analyzed. The angle (direction) at which 21 faces can be recognized.

  FIG. 3B replaces the marker 21 with a screen display so that it is not necessary to attach a card, a sticker, or the like. The marker 21 does not need to be displayed at all times, and may be displayed at the time of initial recognition. In addition, the display without using visible light can eliminate a sense of incongruity.

  Although FIG. 3 shows a case where a recognition area is set by a plurality of markers 21, there may be a case where even one marker 21 can set a recognition area. For example, when the shape of the operation target object 2 is not complicated, is approximately the same size as the marker 21, and may be controlled roughly, the number of the marker 21 is one and a predetermined range around the marker 21 is set as a recognition area. Can do. In addition, by adding information on the size and shape of the recognition area to the information that can be read from the marker 21, the range of the size and shape indicated by the read information is set as the recognition area on the basis of one marker 21. be able to.

  FIG. 4 is a diagram illustrating an example in which a lighting fixture is used as the operation target object 2. In this case, ON / OFF of a lighting fixture, a light quantity, a luminescent color, etc. can be operated by touching the screen of the portable terminal device 1. For example, it is possible to switch ON / OFF by tapping the image of the operation target 2, to increase or decrease the amount of light by dragging up and down on the image, or to change the emission color by dragging left or right. . A marker (21) is provided on the outer surface of the operation target 2 in order to set a recognition area, as in FIG.

  FIG. 5 is a diagram illustrating a configuration example of the mobile terminal device 1.

  In FIG. 5, the mobile terminal device 1 includes a power system 101, a main system 102 including a processor 103, a memory controller 104, and a peripheral interface 105, a storage unit 106, an external port 107, a high-frequency circuit 108, an antenna 109, , An I / O (Input / Output) including an audio circuit 110, a speaker 111, a microphone 112, a proximity sensor 113, a GPS (Global Positioning System) circuit 114, a display controller 116, an optical sensor controller 117, and an input controller 118. ) Subsystem 115, touch-responsive display system 119, optical sensor 120, and input unit 121. The optical sensor 120 corresponds to the camera 11, and the touch-responsive display system 119 corresponds to the screen / touch panel 12.

  FIG. 6 is a diagram illustrating a configuration example of the operation target object control device 3.

  In FIG. 6, the operation target object control device 3 includes a computer device (including a game device) 31 and an interface device 32 as necessary. When the operation target 2 is the large screen monitor shown in FIG. 2, it is sufficient to supply the video / audio signal directly from the computer device 31 to the operation target 2 without providing the interface device 32. When the operation target 2 is the lighting apparatus shown in FIG. 4, an interface device 32 that mediates the exchange of state signals and control signals between the operation target 2 and the computer device 31 is provided.

<Operation>
FIG. 7 is a flowchart showing a processing example of the above embodiment, and shows the processing contents of the mobile terminal device 1.

  In FIG. 7, the mobile terminal device 1 captures the video signal of the camera 11 as data and makes it operable from the program (step S101). For example, this data is obtained by converting a camera image into an aggregate of RGB (Red, Green, Blue) points (pixels). By developing such data on a memory accessible from the program, the program can be operated.

  Next, the mobile terminal device 1 recognizes a plurality of objects (corresponding to the recognition area of the operation object 2) based on the marker 21 (FIG. 3), and obtains basic data necessary for the operation (step S102). The basic data is matrix information about the recognized object, and indicates in which position, at which angle, and in which size the object exists in the three-dimensional space. As an implementation, for example, processing can be performed using an AR library of QUALCOM.

  Next, the mobile terminal device 1 accumulates the basic data and obtains the correlation data while repeating at high speed (step S103). The interrelation data is relative position data between the objects to be recognized or between the object to be recognized and the camera 11.

Next, the mobile terminal device 1 follows real-time changes such as movement of the object based on the correlation data, and corrects camera shake or the like (step S104). As a correction,
-When there are multiple pieces of matrix information for recognizing the object, processing is performed to exclude inaccurate ones. If there is a unique value only at a certain moment while repeating at high speed, processing is performed.

  By continuing to recalculate the analysis of the image and the recognition of the mutual position at high speed (steps S101 to S104), it is possible to follow changes such as movement of the portable terminal device 1 and the operation target 2 at hand.

  Next, when the screen / touch panel 12 is touched by the user, the mobile terminal device 1 performs the following processing.

  First, based on the touch position and the accumulated correlation data after correction, a normal line (normal vector) is virtually extended from the touch position to the back surface of the mobile terminal device 1, and an object exists on the normal line. If it exists, the corresponding operation target position is obtained by obtaining the intersection with the three-dimensional surface of the operation target 2 (step S105). FIG. 8 shows the concept of the processing. The operation target object 2OBJ corresponds to the recognition area of the operation target object 2. When a plurality of objects and normal lines intersect, there are a plurality of operation target positions at this stage.

  Returning to FIG. 7, the mobile terminal device 1 identifies the operation target 2 to be the operation target from the operation target position and the effective range of the target (step S106). That is, when each of the plurality of objects has an effective range (in the case of a screen, a rectangle), the operation object 2 is specified by determining which one should be preferentially operated. As a determination criterion, priority is given to the vicinity (closer to the portable terminal device 1) as a standard.

  Next, the mobile terminal device 1 converts the operation target position into the coordinates of the operation target object 2 (step S107). That is, the part where the normal line for operation intersects the specific shape of the target object is obtained, and when the shape is a solid having a plurality of faces, the intersection of each of the closed faces Evaluate and transform coordinates. In the three-dimensional shape, there are cases where intersections are established with respect to a plurality of surfaces, but the vicinity is given priority. Which of the plurality of intersecting points is to be preferentially operated (priority is given priority in principle) is determined, and local matrix information of the target device is generated.

  Next, the mobile terminal device 1 calculates local data for controlling the operation target 2 based on the converted coordinates (step S108).

  Next, processing when an event such as tap, drag and drop, or flick is established will be described. The establishment of the event is determined as follows.

  The tap is determined by touching or separating the screen / touch panel 12 of the mobile terminal device 1.

Drag and drop
・ Drag while moving the finger while fixing the object and hand terminal ・ Drag while moving the hand terminal while fixing the object and finger ・ Drag while moving the object while holding the hand terminal and finger 3 Dragging occurs when two patterns are generated, and dropping is determined when the finger is released.

  The second pattern means that drag-and-drop or the like can be realized by moving the hand device or the target device without moving the fingertip while tapping.

  In addition, the case where two or more types of related information (object coordinates, hand terminal coordinates, finger position) move simultaneously is also established.

  The flick is valid when an operation corresponding to the flick is established on the touch panel of the hand apparatus. This is because it is naturally accepted that the target device is affected by the distance and angle and input is difficult, and that it is established by the touch of the hand device. The determination itself of whether or not a flick has been established at the hand device is left to the OS or driver of the hand device. This is to avoid creating a sense of incongruity by using the same conditions as are used normally.

  Flick determination is based on the condition that the slide distance on the touch panel is equal to or longer than the predetermined distance and the time taken for the slide is within a predetermined time. Is determined to have been performed on the touch panel.

  In FIG. 7, when the event is established, the mobile terminal device 1 generates a message in which its own device ID is added to local data (step S109).

  Next, the mobile terminal device 1 transmits a message to the operation object control device 3 that controls the operation object 2 (step S110). The message may be data equivalent to a mouse, but since the number of gesture patterns that can be input at the local terminal is expected to increase, the message system is a general-purpose message system that leaves an undefined area that can be expanded.

  The operation target control device 3 that has received the message performs processing corresponding to the message and reflects the result on the operation target 2. When the operation object 2 is a large screen monitor (FIG. 2), operations such as tap, drag and drop, flick, etc. are applied to buttons, icons and characters displayed on the large screen monitor, and the operation process and the resulting image are displayed. Display on a large screen monitor. When the operation target 2 is a lighting fixture (FIG. 4), the power ON / OFF, the light amount, the emission color, and the like are controlled according to the operation content.

  Further, a response message may be received from the operation target object control device 3 after the message transmission (step S110), and a process corresponding to the received message may be performed.

  In the above embodiment, the input operation to the touch panel has been mainly described. However, the recognized operation object and the image linked to the operation can be superimposed and displayed on the touch panel.

  In the present embodiment, since the position information of the object in the virtual three-dimensional space can be used, for example, the intersection with the recognition area (coordinate position of the target device) detected based on the operation on the touch panel flies from the touch panel. When the coordinate position of the target device that the user is trying to operate is indicated by compositing and displaying an image of the object to be moved (such as an arrow or a ball), the object moves in the space containing the target object. Can be displayed. Thereby, the user can operate more intuitively.

  In addition, for example, when an image of an object indicating an input result (such as a balloon indicating hit or miss) is displayed around the target based on an operation on the touch panel, the image is displayed according to the position of the target. Can do. FIG. 9 shows an example in which an image 202 such as a title is displayed along the upper end of the image 201 of the operation target 2 on the screen of the mobile terminal device 1 and a character image 203 is displayed on the left side.

<Summary>
As described above, according to this embodiment, it is possible to intuitively operate a wide range of devices in consideration of a three-dimensional arrangement.

  The present invention has been described above by the preferred embodiments of the present invention. While the invention has been described with reference to specific embodiments, various modifications and changes may be made to the embodiments without departing from the broad spirit and scope of the invention as defined in the claims. Obviously you can. In other words, the present invention should not be construed as being limited by the details of the specific examples and the accompanying drawings.

DESCRIPTION OF SYMBOLS 1 Portable terminal device 11 Camera 12 Screen / touch panel 101 Power supply system 102 Main system 103 Processor 104 Memory controller 105 Peripheral interface 106 Memory | storage part 107 External port 108 High frequency circuit 109 Antenna 110 Audio circuit 111 Speaker 112 Microphone 113 Proximity sensor 114 GPS circuit 115 I / O subsystem 116 Display controller 117 Optical sensor controller 118 Input controller 119 Touch-responsive display system 120 Optical sensor 121 Input unit 2 Operation object 21 Marker 2OBJ Operation object object 3 Operation object control device 31 Computer device 32 Interface device

Claims (5)

A display unit provided on the front surface of the housing and a touch panel integrated with the display unit;
A camera provided on the back surface of the housing, and having a field of view in the direction of the back surface and normal of the display unit;
Display means for displaying a captured image of the camera on the display unit in real time;
Position information calculation means for constructing a virtual three-dimensional space based on a captured image of the camera, detecting that the target image is included in the captured image, and calculating position information in the virtual three-dimensional space; ,
When there is a touch input on the touch panel, an intersection that detects an intersection between a normal vector passing through the touch input position on the touch panel and a recognition area set in advance in the object positioned in the virtual three-dimensional space Detection means;
A portable terminal device comprising operation message transmission means for transmitting an operation message to the object control device based on the detected position of the intersection. The mobile terminal device according to claim 1,
The object is an image display device having a screen to be a recognition area,
The said operation message transmission means produces | generates the positional information on the intersection in a recognition area, and transmits to the said control apparatus, when the said intersection is detected. In the portable terminal device according to any one of claims 1 and 2,
A determination unit configured to determine that a specific operation has been performed on the touch panel based on a determination condition set for the operation of the touch panel;
The operation message transmission unit adds the determination result of the determination unit to the position information of the object and transmits it to the control device. The mobile terminal device according to claim 3.
The determination means determines that a flick input has been performed on the touch panel on the condition that the slide distance on the touch panel is equal to or greater than a predetermined distance and the time taken for the slide is within a predetermined time. The portable terminal device characterized by this. A display unit provided on the front surface of the housing, a touch panel integrated with the display unit, and a camera provided on the back surface of the housing and having a field of view in the direction of the back surface and the normal line of the display unit. A computer constituting the mobile terminal device;
Display means for displaying the captured image of the camera on the display unit in real time;
A position information calculation means for constructing a virtual three-dimensional space based on a captured image of the camera, detecting that the captured image includes an object, and calculating position information in the virtual three-dimensional space;
When there is a touch input on the touch panel, an intersection that detects an intersection between a normal vector passing through the touch input position on the touch panel and a recognition area set in advance in the object positioned in the virtual three-dimensional space Detection means,
A portable terminal device control program that functions as an operation message transmission unit that transmits an operation message to the target control device based on the detected position of the intersection.