JP2012155480A - Input device for portable information apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input device for a portable information apparatus for enabling input operation by a simple action even when the portable information device is held by both hands or when the device is held by one hand and the other hand cannot be used due to being used for another thing.SOLUTION: An input device for a portable information apparatus uses a three-dimensional space and integrates a three-dimensional distance sensor 3 with a display device 2 of the portable information device 1. The input device uses three-dimensional position information on a target object detected by the three-dimensional distance sensor 3, for example a part of a human body or an object therewith such as a face, nose, chin, hands, and fingers existing near the display device 2, uses two-dimensional position information on the target object in lateral and longitudinal directions viewed from the three-dimensional distance sensor 3 as two-dimensional indication position information in the display device 2, and uses position information on the target object in a distance direction as information for determining timing of optional operation execution.

Description

  The present invention relates to an application technology of a three-dimensional distance sensor that can be realized at low cost by a parallax sensor LSI that has already been developed (see Patent Documents 8 to 10 and Non-Patent Documents 1 to 3). The present invention relates to an input device for a portable information device using a distance sensor.

  In general, a keyboard and a mouse are mainly used for operation input to a portable information device such as a personal computer. Character (character or number) information can be input by pressing a corresponding button with a finger or the like on the keyboard. Further, the mouse can freely move the two-dimensional indication position in the display device by the movement on the plane, and the timing for executing any operation can be determined by a button or the like integrated with the mouse. In televisions, audio devices, and the like, desired functions can be input at hand by using an infrared communication type remote control device equipped with a large number of buttons for each operation function. In any of these input methods, it is necessary to operate a button or a dedicated device provided with a movement detection function separately from the portable information device with a hand or a finger.

On the other hand, in a portable information device such as a smartphone or a tablet terminal, a small button group provided in the device or a touch panel integrated with a display device is operated with a finger or the like to perform input. These are devised so that a desired input can be performed with a small number of buttons and operations using an operation input prediction function or the like so that the portable terminal can be operated without stress even in a small area. However, any portable information device requires at least one hand for the input operation. Since portable information devices are usually held by hand, both the holding hand and the operating hand are blocked, making it difficult to operate while holding other objects or performing other operations. . In addition, there is a problem that a size such as a tablet terminal that is difficult to hold stably with one hand cannot be operated unless it is placed somewhere.
In the future, in order to further improve the convenience of the portable information device, it is necessary to devise an input method.

  The input device using the three-dimensional space according to the present invention was invented under such a background, and by applying the three-dimensional distance sensor that can be realized at low cost by the previously disclosed parallax sensor LSI, portable information It facilitates the input operation of the equipment and solves the conventional problems.

  A method using image data captured by a camera as an input device such as a personal computer has been proposed (see, for example, Patent Documents 1 to 5). They detect eyes, pupils, irises, mouths, and the like in a face from an image captured by a camera by using a pattern matching method or the like, and use the two-dimensional position information or the change information of the shape as 2 A common feature is that it is used as dimension indication position information or some operation information. Unlike the present invention, these methods use a two-dimensional camera image and detect changes in the position and shape of the object using a pattern matching method. In order to achieve this, it is necessary to perform various image adjustment processes and pattern matching processes in large quantities, and there is a problem that apparatus cost and processing time increase. Further, the present invention is different from the present invention in that no distance information with respect to the object is used.

  In addition, there has been proposed a method of moving a cursor or the like in a display device by tilting the device using a large number of acceleration sensors built in the mobile terminal (see, for example, Patent Document 6). However, generally used acceleration sensors have low detection sensitivity to changes in angle, and there is a problem that sufficient static angle accuracy cannot be obtained simply by tilting the apparatus. Therefore, in general, a method of moving the cursor or the like with an expression of force in that direction proportional to the magnitude of the inclination is taken. In this method, there is a problem that it is difficult to place the cursor accurately at a desired place. Even if the sensitivity of the acceleration sensor can be sufficiently increased at the expense of cost, there is a problem that the gravitational acceleration for detecting the tilt angle cannot be separated in principle from the other acceleration applied to the device itself, and is moving. The problem that it becomes difficult to use in situations that are frequently encountered in daily life, such as cases or when the posture changes.

  In addition, an apparatus that detects a three-dimensional position from a stereo image and outputs an image in the direction indicated by the detected hand has been proposed (Patent Document 7), but the apparatus configuration requirements are different from the present invention.

JP 2006-59147 A JP 2006-309291 A JP 2007-102415 A Japanese Patent Application Laid-Open No. 2004-362569 JP-A-9-305743 Special table 2007-509448 JP 2003-141510 A JP 2005-265457 A JP 2009-236661 A JP 2010-197227 A

T. Yoshida and Y. Arima, "Binobjective monocular optical module for single-chip stereo vision sensor LSI," IEICE Electronics Express, Vol. 3, No. 17, pp. 390-396, Sep. 10, 2006. Masatomo Kawano, Norihito Kawaguchi, Takashi Yoshida, and Yutaka Arima, "Three-Dimensional Binocular Range Sensor LSI with Enhanced Correlation Signal," Japanese Journal of Applied Physics, Vol. 49, No. 4D, pp. 04DE05-1-6, Published April 20, 2010. Norihito Kawaguchi, Masatomo Kawano, and Yutaka Arima, "Three-Dimensional Binocular Range Sensor Large Scale Integration with a 410μs / Frame Output Time High-Speed Data Output Method," Japanese Journal of Applied Physics, Vol. 49, No. 4D, pp 04DE06-1-4, Published April 20, 2010.

  An input device provided in a portable information device such as a smartphone is a large number of buttons, a touch panel, and the like displayed on the screen of a display device, which need to be operated with a finger or the like. Usually, one hand is used to hold the portable device, and both hands are required to operate the device together with the operating hand. For this reason, there is a problem that it is difficult to operate while holding another object or performing other work, and the convenience as a portable device is impaired. Therefore, it is desired to develop an apparatus that can perform an input operation by a simple method without using a hand.

  Also, for portable devices that are difficult to hold stably with one hand, such as a tablet device, input operations cannot be performed if the device is held with both hands, so the device should be placed on a desk or knee. Become. This restriction greatly impairs the use environment and use opportunity, and narrows the use range of the device. Therefore, development of an input device that can be easily operated while being held with both hands is desired.

  The present invention is for a portable information device in which an input operation can be performed with a simple operation even when the portable information device is held with both hands, or even when the device is held with one hand and the other hand is blocked. An object is to provide an input device.

In order to solve the above-described problems, the present invention proposes a method of inputting using an object that always exists in the vicinity of the device in addition to the hand in consideration of the use environment of the portable information device.
A first configuration of the present invention is an input device for a portable information device that uses a three-dimensional space. The three-dimensional distance sensor and the display device of the portable information device are integrated and detected by the three-dimensional distance sensor. An input device for a portable information device that determines two-dimensional position information in the display device and execution timing of an arbitrary operation based on the three-dimensional position information of the target object.

  In this first configuration, the three-dimensional distance sensor integrated with the portable information device detects the three-dimensional position of the object, and the display device is based on the two-dimensional position information in the vertical and horizontal directions viewed from the sensor. And the execution timing of an arbitrary function corresponding to the button function is determined based on the distance direction information. Specifically, the detected object can be moved relatively by tilting or moving the device slightly with the hand holding the portable information device, with high accuracy (sensitivity). The position can be set statically. Here, “two-dimensional position information” refers to position information represented by general two-dimensional coordinate data or the like.

According to a second configuration of the present invention, a switch function for determining the execution timing of some operation is installed in the display device or in the portable information device. Device.
In the second configuration, it is possible to perform switch operations of various functions by pressing an actual switch or a switch screen displayed in the display device with a finger of a hand holding the portable information device. Become.

According to a third configuration of the present invention, as a target to be detected by the three-dimensional distance sensor, a human body or an object associated therewith, such as a human face, nose, chin, hand, or finger existing near the display device And the two-dimensional position information in the horizontal and vertical directions viewed from these three-dimensional distance sensors as the two-dimensional designated position information in the display device, and the distance direction This position information is used as information for determining the execution timing of an arbitrary operation. This is an input device for a portable information device having the first or second configuration.
In this third configuration, by using a part of the human body such as the head, face, nose, and chin of the person who is viewing the display screen of the device as an object, without using a special instrument, In addition, various input operations such as scrolling the screen and turning pages can be performed without performing an operation such as tracing the display screen with a hand or a finger. Here, “two-dimensional indication position information” refers to position information represented by two-dimensional coordinate data of a means for indicating the position of an image such as a cursor. The “position information in the distance direction” refers to position information represented by coordinate data of a distance (depth) detected by a three-dimensional distance sensor.

According to a fourth configuration of the present invention, a part of a human body such as a human face, nose, and jaw detected by the three-dimensional distance sensor by tilting the integrated three-dimensional distance sensor and the display device. For the portable information device of the third configuration, wherein the three-dimensional position of the display device is relatively changed, and the information for determining the execution timing of the two-dimensional indication position information and arbitrary operation in the display device is operated. It is an input device.
In the fourth configuration, a part of the human body is moved relatively from the three-dimensional distance sensor not by moving a part of the human body but also by tilting or turning the display device up and down, left and right. Therefore, an input operation similar to that of the third configuration can be performed.

According to a fifth configuration of the present invention, two-dimensional position information in the display device is arbitrarily selected using only three-dimensional position information of an object detected by the three-dimensional distance sensor. This is an input device for a portable information device having a third or fourth configuration characterized by determining the execution timing of the operation.
In this fifth configuration, among a plurality of points such as a human face, nose and chin, an object having the closest distance from the three-dimensional distance sensor is determined by the coordinates of the plurality of points, and the movement of the point is detected. By doing so, the input operation can be facilitated.

  According to the present invention, for a portable information device that can be held with one hand such as a smartphone, an input operation can be performed with only one hand that is held, while holding other things or doing other work. Operation becomes possible. In addition, devices that need to be held with both hands, such as a tablet terminal, can be input while being held with both hands, and can be used in various situations. For these reasons, the use of the portable information device is further facilitated, and the respective fields of use can be expanded.

1 is a front view of a portable information device according to an embodiment of the present invention. It is a front view which shows the example which implement | achieved the function button with the switch in the portable information device which concerns on embodiment of this invention. In the portable information device according to the embodiment of the present invention, it is a front view showing an example in which function buttons are realized by switch icons displayed in a display device. It is a front view which shows the state which held the portable information device which concerns on embodiment of this invention with both hands. It is a perspective view which shows the state which hold | maintained the portable information device which concerns on embodiment of this invention with both hands, and was close to the face. It is a block diagram which shows the circuit structure of the portable information device which concerns on embodiment of this invention. It is a flowchart which shows the example of the process by the portable information device which concerns on embodiment of this invention. It is a figure which shows the principle of the distance detection by a binocular parallax.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  A configuration example of a portable information device according to the present invention is shown in FIG. The portable information device 1 is characterized by a device configuration in which a display device 2 and a three-dimensional distance sensor 3 mounted on the portable information device 1 are integrated. The three-dimensional distance sensor 3 is installed in such a direction that the face of a person who looks at the display device 2 falls within the field of view. Alternatively, it can be fixed as such. Since the three-dimensional distance sensor 3 detects and outputs three-dimensional position information using a parallax sensor including two imaging elements (image sensors), it is necessary to acquire images from two different positions. Two lens holes 4 for stereo viewing as shown in FIG.

FIG. 8 is a diagram for explaining the principle of distance detection by two image sensors. As shown in FIG. 8A, an image pickup device corresponding to the right eye and an image pickup device corresponding to the left eye are set apart from each other and imaged. Here, an image captured by an image sensor corresponding to the left eye (hereinafter referred to as “left eye image”) is represented by {a ^(L) _{i, j} | i = 1, 2,..., N, j = 1, 2, ..., m}. An image captured by an image sensor corresponding to the right eye (hereinafter referred to as a “right eye image”) {a ^(R) _{i, j} | i = 1, 2,..., N, j = 1, 2,. , M}.

With respect to the same object, the positions of the objects imaged by the left and right imaging elements are shifted according to the distance from the imaging element to the object. Therefore, when considering only the horizontal correlation between the captured left-eye image and right-eye image, the correlation between all (a ^(L) _{i, k} , a ^(R) _{j, k} ) pairs is calculated. If so, the distance of each object can be detected by the coordinates represented by the set having the largest correlation.

  In FIG. 8A, the three circles A, B, and C indicate the object. When these objects are imaged by the left and right imaging elements to create a correlation matrix between the left eye image and the right eye image, the result is as shown in FIG. In FIG. 8B, it is assumed that there is a correlation function at the position of the intersection of the lines 1 to n of the left eye image and the lines 1 to n of the right eye image. A large correlation is detected with respect to the objects A, B, and C in FIG. 8A at the three circle positions A, B, and C shown in FIG. Therefore, if a coordinate having a large correlation is detected on the correlation matrix, and the coordinate is converted into an oblique coordinate in FIG. 8A, the distance to the object can be detected. Note that the oblique coordinates shown in FIG. 8A can be determined by the positions of the left and right imaging elements and their relative angles. Therefore, the coordinate conversion from the correlation matrix to the oblique coordinates can be easily performed by referring to a conversion table prepared in advance.

  In the distance detection by the stereo method described above, a large-scale calculation is generally required for the parallax image calculation processing of the two captured left and right images. In the present embodiment, the three-dimensional distance sensor 3 can be realized at a high speed, a compact size and a low price by using the previously disclosed parallax sensor LSI (see Patent Documents 8 and 9).

  That is, the parallax sensor LSI captures an imaging target and outputs the first image as a pixel signal that is an analog voltage signal, and the imaging target from an angle different from the first imaging element. A second image sensor that outputs a second image as a pixel signal that is an analog voltage signal, and each of the pixel signals output from the first and second image sensors is a voltage value of each pixel signal. A plurality of voltage / pulse width conversion circuits for converting to a pulse width pixel signal having a pulse width of a length proportional to the same, and all the voltage / pulse width conversion circuits simultaneously convert the pixel signal to a pulse width pixel signal. A synchronous control circuit that performs timing control so as to convert, and two pulse width pixel signals each composed of a combination of each pulse width pixel signal of the first image and each pulse width pixel signal of the second image. In contrast, a plurality of correlation detections that convert the total pulse length of the differential pulse obtained by taking the exclusive OR of both into a signal of voltage value or current value proportional to the total pulse length and output this signal as a correlation signal And a circuit and an element including the circuit are highly integrated.

  According to this parallax sensor LSI, a pixel signal that is an analog voltage signal is temporarily mapped to the time axis of the pulse width signal by the voltage / pulse width conversion circuit, and the correlation detection circuit performs a logical operation on the voltage axis on the time axis. In addition, the parallax image calculation process can be performed at high speed by a small-scale circuit by returning this to the analog voltage signal or current value and using it as a correlation signal. Accordingly, it is possible to execute a stereo image correlation operation at high speed and output a parallax image. In addition, both the circuit area and the power consumption can be made smaller than before.

  In the present invention, the three-dimensional position information of the detected object output from the three-dimensional distance sensor 3 is an arithmetic device in a portable information device or an ASIC (Application Specific Integrated Circuit) provided in the vicinity of the three-dimensional distance sensor 3. In the information processing apparatus such as the above, the information is processed into two-dimensional indication position information such as a cursor displayed in the display device 2 and operation information of an arbitrary function switch. For example, using only the three-dimensional position information of the closest object among the objects detected by the three-dimensional distance sensor 3, the two-dimensional position information in the vertical and horizontal directions is directly used as position information such as a cursor, and the distance information is functioned. There is a method used for switch operation information. In this case, paying attention to each distance data from a plurality of detected three-dimensional position data sets output from the three-dimensional distance sensor 3 for each frame, only the object having the smallest value is extracted, The two-dimensional position data is used as position data for the cursor C in the display device 2 after being multiplied by a constant. Further, using the amount of change in the nearest distance data between frames, for example, when the amount of change is larger than a certain value, a process of outputting a signal for executing a function switch operation may be executed.

  In addition, a method of similarly using an average value of three-dimensional data of an object detected in a space within a certain range from the position of the object at the closest distance may be considered. In any processing method, since the information processing can be executed with a relatively simple and small number of processes, there is a feature that a large load and cost are not applied to the arithmetic unit of the main body or the dedicated processing unit.

  As shown in FIG. 2, a function button 5 is provided on the device itself, or, as shown in FIG. 3, the button function 6 is displayed in the display device 2 by using the touch panel function. There is also a method of operating the switch without changing the relative distance between the two. This enables quick switch operation.

  4 and 5 show an example of a situation in which an input operation such as movement of the cursor C in the display device 2 is performed by holding the portable information device 1 with both hands and tilting it. .

  Hereinafter, an example of processing by the portable information device according to the embodiment of the present invention will be described with reference to FIGS. 6 and 7.

  FIG. 6 shows a circuit configuration of a portable information device (hereinafter simply referred to as “information device”) 1 according to an embodiment of the present invention. The memory 110 and the same chip in this example are shown. 120, a memory controller 121, a CPU 122, an interface 123, a communication unit 160 including an RF (Radio Frequency) circuit 161 and an antenna 162, an audio unit 170 including an audio circuit 171, a speaker 172, and a microphone 173, and a display An input / output (I / O) subsystem 130 comprising a controller 131 and other input controllers 132, a touch panel 140 constituting a display device, other input devices 150, a three-dimensional distance sensor 181 and a two-dimensional position calculation unit. And a space pointing device 180 composed of 182. .

The memory 110 includes a high-speed random access memory and other memory devices, and stores an OS (Operating System) and various applications.
Access to the memory 110 by other components of the information device 1, for example, the CPU 122 and the interface 123, can be controlled by the memory controller 121.
The interface 123 couples the input and output peripheral devices of the information device 1 to the CPU 122 and the memory 110.

The RF circuit 161 receives and transmits an RF signal. The RF circuit 161 communicates with a communication network or another communication device using an electromagnetic wave signal. The RF circuit 161 can communicate with a network such as the Internet, a mobile phone network, a wireless local area network (LAN), or the like by wireless communication.
The audio circuit 171, the speaker 172, and the microphone 173 constitute an audio interface between the user and the information device 1.

The I / O subsystem 130 couples input / output peripheral devices of the information device 1, for example, the touch panel 140 and other input devices 150 to the interface 123. The I / O subsystem 130 may include a display controller 131 for one or more other input devices 150 and one or more other input controllers 132. Other input devices 150 can include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like.
The touch panel 140 is used to implement virtual or soft buttons and one or more soft keyboards.

  The spatial pointing device 180, which is a feature of the present invention, includes a three-dimensional distance sensor 181 and a two-dimensional position calculation unit 182, for example, a three-dimensional distance sensor function using the parallax sensor LSI disclosed in Patent Documents 8 and 9 described above. However, the present invention is not limited to this, and a three-dimensional distance sensor and a two-dimensional position calculation unit using other methods can also be used. The two-dimensional coordinates of the specific object output from the space pointing device 180 are input to the display controller 131, and operation information similar to the movement indicated by the finger or the like on the touch panel 140 is obtained by the movement of the two-dimensional coordinates. Are transmitted to the interface 123. In the present embodiment, the space pointing device 180 is integrated with the information device 1, attached to the frame of the information device 1, and a connector (not shown) such as a USB port provided on the frame of the information device 1. Can be integrated as an external device. In that case, the spatial pointing device 180 may be connected to the other input controller 132 as the other input device 150.

A specific operation information processing procedure is shown in FIG.
First, for example, when the three-dimensional distance sensor displayed on the touch panel 140 is turned on, as shown in S100, the three-dimensional distance sensor 181 in the space pointing device 180 is activated, and each point of the object obtained from the left and right sensors is operated. The three-dimensional coordinates from the origin (the position of the midpoint of the left and right sensors) are obtained by calculation.

In step S110, the two-dimensional position of the point (the operation point, for example, the tip of the nose) that is closest to the origin among the obtained three-dimensional coordinates of each point is displayed as a cursor on the touch panel.
In step S120, the cursor is moved on the touch panel by tilting or swinging the hand holding the information device 1 or moving the operation point while the information device 1 is fixed by hand.
In step S130, the function buttons 5 and 6 are pressed with a finger, a function switch is operated by a change in distance data detected by the space pointing device 180, or the cursor C Various methods are conceivable, for example, by detecting that the position of the sensor does not change greatly for a certain period of time.

In step S140, an output is output to the display controller 131 so as to perform a preset operation according to the pattern of the two-dimensional trajectory to which the cursor has moved. For example, if the two-dimensional trajectory to which the cursor moves is vertical, the screen displayed on the touch panel 140 is scrolled up and down. If the two-dimensional trajectory is horizontal, the screen is scrolled left or right or page is turned. If it is circular, screen operations such as enlarging and reducing the screen are performed depending on whether it is clockwise or counterclockwise.
In step S150, when the next operation button is pressed, the process returns to step S120 to continue the processing. Otherwise, the process is terminated.

  As described above, according to the present embodiment, a portable information device that can be held with one hand, such as a smartphone, can be input with only one hand that is held, and can have other items. It is possible to perform operations while In addition, devices that need to be held with both hands, such as a tablet terminal, can be input while being held with both hands, and can be used in various situations. For these reasons, the use of the portable information device is further facilitated, and the respective fields of use can be expanded.

  The present invention is a user interface technology using relative three-dimensional position information of an object that can generally exist in a three-dimensional space, and can be used for an input device for a portable information device such as a smartphone or a tablet terminal. .

DESCRIPTION OF SYMBOLS 1 Portable information device 2 Display apparatus 3 Three-dimensional distance sensor 4 Stereo vision lens hole 5 Function buttons (switches)
6 Function buttons (displayed on the display device)
7 Hand Holding Portable Information Device 8 Head of Person Looking at Display Device 9 Nose of Person Looking at Display Device 10 Jaw of Person Looking at Display Device 110 Memory 120 Chip 121 Memory Controller 122 CPU
123 Interface 130 I / O Subsystem 131 Display Controller 132 Other Input Controller 140 Touch Panel 150 Other Input Device 160 Communication Unit 161 RF Circuit 162 Antenna 170 Audio Unit 171 Audio Circuit 172 Speaker 173 Microphone 180 Spatial Pointing Device 181 3D Distance Sensor 182 2D position calculator

Claims (5)

  An input device for a portable information device using a three-dimensional space, in which a three-dimensional distance sensor and a display device of the portable information device are integrated, and three-dimensional position information of an object detected by the three-dimensional distance sensor is obtained. An input device for a portable information device, wherein two-dimensional position information in the display device and execution timing of an arbitrary operation are determined based on the information.   2. The input device for a portable information device according to claim 1, wherein a switch function for determining execution timing of some operation is provided in the display device or in the portable information device.   As an object to be detected by the three-dimensional distance sensor, three-dimensional position information of a human body or a part of an object associated therewith such as a human face, nose, chin, hand, and finger existing near the display device. The two-dimensional position information in the horizontal and vertical directions as viewed from the three-dimensional distance sensors is used as the two-dimensional designated position information in the display device, and the position information in the distance direction is used to execute an arbitrary operation. The portable information device input device according to claim 1, wherein the input device is used as information for determining timing.   By tilting the integrated three-dimensional distance sensor and the display device, the three-dimensional position of a part of the human body such as the face, nose, and jaw detected by the three-dimensional distance sensor is relatively changed. 4. The portable information device input device according to claim 3, wherein two-dimensional indication position information in the display device and information for determining an execution timing of an arbitrary operation are operated.   Of the objects detected by the three-dimensional distance sensor, only the three-dimensional position information of the object having the closest distance is used to determine the execution timing of the two-dimensional position information in the display device and an arbitrary operation. The input device for portable information equipment according to claim 3 or 4.