WO2019003747A1

WO2019003747A1 - Wearable terminal

Info

Publication number: WO2019003747A1
Application number: PCT/JP2018/020283
Authority: WO
Inventors: 軌行石井
Original assignee: コニカミノルタ株式会社
Priority date: 2017-06-26
Filing date: 2018-05-28
Publication date: 2019-01-03

Abstract

Provided is a wearable terminal with which it is possible to easily perform input via a simple input member. A wearable terminal, worn on the head of a user, for performing input of information via an input member which forms at least two or more kinds of characters, numerals, and/or symbols in a delimited region each and with which it is possible to indicate the delimited region by an input indicator used by the user. The wearable terminal has a display unit for displaying an image in such a way as to be visible to the user, a camera for imaging a subject and outputting an image signal, a processing unit for processing the image signal outputted from the camera, and a storage unit for storing the feature of the image of the input member. The processing unit: starts execution of an input mode when it is determined, on the basis of the image signal of the subject outputted from the camera and the feature of the image of the input member stored in the storage unit, that the subject imaged by the camera is at least a portion of the input member; and inputs information that corresponds to the characters, numerals, and/or symbols formed in the indicated delimited region when it is determined, while the input mode is being executed, that the input indicator has indicated one of the delimited regions of the input member.

Description

Wearable terminal

The present invention relates to a wearable terminal capable of performing input via an input member.

Wearable terminals such as head-mounted displays (hereinafter referred to as HMDs), which have been rapidly developed in recent years, do not hold the display by hand because the display comes in front of the user by being attached to the head of the user. The information displayed can be viewed by the user, and is easy to carry and handle.

By the way, when working while wearing the HMD and looking at the display, it is necessary to enter letters, numbers, etc., such as "Enter password at user login" and "Enter search string". There is. On the other hand, for example, a keyboard widely used as a user interface is connected to the HMD via a wire or wireless such as Bluetooth (registered trademark), and characters and numbers are input by operating the keyboard. It is possible. However, since the keyboard contains a large number of keys corresponding to letters and numbers, it is generally bulkier than a wearable terminal, and if it must be held for input, portability is an advantage of the wearable terminal. There is a risk of damaging

Furthermore, when trying to connect the HMD and the keyboard via a wire, the wiring becomes an obstacle, which hinders the input, impairs the ease of handling, or increases the cost because it is necessary to add a connector. On the other hand, when trying to connect the HMD and the keyboard via radio, operations such as pairing to recognize each other are required in advance, and when trying to share one keyboard with a plurality of wearable terminals, Mistakes may occur due to connection contention and the like. On the other hand, a new technology has been developed: a virtual keyboard that allows the user to input in response to the image emitted from the projector. However, to introduce such technology, the projector is first required. This increases the size, impairs portability, and increases costs.

On the other hand, Patent Document 1 discloses a technique of analyzing the position of a finger captured by a camera in the HMD, and controlling the input by associating the position information with the position information of the button displayed on the display.

JP, 2016-31761, A

However, according to the technology of Patent Document 1, there is a disparity between the reference screen and the operation screen due to the presence of parallax according to the distance between the user's line of sight and the optical axis of the camera. In order to correct the deviation, it is necessary to analyze the finger position in a certain coordinate system and match it with the display coordinate system. However, for the matching, complex arithmetic processing must be performed at high speed, and a high-performance CPU or the like is required, which may significantly increase the cost of the wearable terminal.

The present invention has been made in view of the above-described circumstances, and an object of the present invention is to provide a wearable terminal capable of performing input easily through a simple input member.

In order to realize at least one of the above-mentioned objects, a wearable terminal reflecting one aspect of the present invention is mounted on the head of the user and divided into at least two or more types of characters, numbers and / or symbols. A wearable terminal which is formed in the inside and which inputs information through an input member which can designate the divided area with an input indicator used by a user,
A display unit that displays an image so that the user can view it;
A camera which picks up an object and outputs an image signal;
A processing unit that processes an image signal output from the camera;
A storage unit for storing features of an image of the input member;
The processing unit is configured such that a subject captured by the camera is at least a part of the input member based on an image signal of the subject output from the camera and a feature of the image of the input member stored in the storage unit. When it is determined that the input mode is started, execution of the input mode is started, and when it is determined that the input indicator has instructed any of the divided areas of the input member while the input mode is being executed, Information corresponding to the characters, numbers and / or symbols formed in the designated division area is inputted.

According to the present invention, it is possible to provide a wearable terminal capable of performing input easily through a simple input member.

It is a perspective view of HMD100 which constitutes a wearable terminal concerning this embodiment. It is a block diagram of HMD100 concerning this embodiment. It is a figure which shows a keyboard. It is a figure which shows the state which inputs using a keyboard KB using HMD100 concerning this embodiment. It is a flowchart which shows control which performs input via the keyboard KB using HMD100. FIG. 6 is a diagram showing an example of an image displayed on the image display unit 104. It is a figure which shows the input member concerning a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a perspective view of an HMD 100 constituting a wearable terminal according to the present embodiment. FIG. 2 is a block diagram of the HMD 100 according to the present embodiment.

As shown in FIG. 1, the HMD 100 has a frame 101. The rectangular frame 102 is attached to a lower end of the front side of the frame 101 substantially U-shaped as viewed from above, and the main body portion 103 is attached to the upper part thereof. The main body unit 103 includes an image display unit 104, a camera 105 for imaging an object, and a microphone 106 which is a sound acquisition device. The main unit 103 is connected to the control box CB by the wiring CD, and is supplied with power from a battery (not shown) built in the control box CB. With the frame 101 mounted on the head of the user, a rectangular frame 102 is placed in front of the user's right eye. In the rectangular frame 102, an image display unit 104 capable of through display is disposed.

In FIG. 2, the control box CB includes a control unit (processing unit) 107 and a memory (storage unit) 108. The memory 108 is attached to its shape, which is an inherent feature of the keyboard KB (see FIG. 3), and its shape, which is a feature of the user's finger FG (see FIG. 4) functioning as an indicator. And the characteristics of the unique sound (keying sound) emitted when the key KY of the keyboard KB is pressed. The control unit 107 has a function of receiving an image signal output from the camera 105 and performing image processing and pattern matching to be described later.

FIG. 3 is a diagram showing a keyboard. Although the keyboard KB as an input member is an actual keyboard here, it is used without being connected to the HMD 100 even in that case. A general keyboard KB is standardized (alphabet arrangement, JIS arrangement etc.), and different types of characters, numbers and symbols (hereinafter referred to as characters) are respectively formed in the center of the key (division area) KY doing. However, instead of the keyboard KB, a toy keyboard or a printed picture or illustration of a keyboard may be used as an input member, thereby improving portability. These are collectively referred to as "an object shaped like a keyboard". Further, the “partitioned area” refers to, for example, a range recognized as being surrounded by an edge formed around a character or the like in an image acquired by imaging. Furthermore, “to indicate with a pointer” includes, for example, the overlapping of the tip of the pointer with the sectioned area in an image acquired by imaging, and in this case, performing key operation with a finger.

Next, the operation of this embodiment will be described. FIG. 4 is a diagram showing a state in which an input is performed via the keyboard KB using the HMD 100 according to the present embodiment, and is shown in the state of an image captured by the camera 105. As shown in FIG. 4, here, a seal mark MK of a red triangle (but not limited to this color and shape) is attached to the nail of the user's index finger FG. Such a seal mark MK is easy to pattern match, which will be described later, and can be attached, for example, to the finger of a gloved hand, so that even a user working with a glove can input. However, input can also be performed without using the seal mark MK.

FIG. 5 is a flowchart showing control for performing input via the keyboard KB using the HMD 100. FIG. 6 is a diagram showing an example of an image displayed on the image display unit 104. As shown in FIG. First, it is assumed that the HMD 100 operates and the camera 105 captures an image of a subject, and the control unit 107 processes an image signal from the camera 105 and performs pattern matching as needed. However, at this point in time, the input mode has not yet been executed, and the HMD 100 can execute only the minimum functions, and in order to realize higher functions, password input or the like is required.

Note that “pattern matching” refers to a model from a reference image by applying an edge extraction method to a reference image (the shape of the keyboard KB stored in the memory 108) and a subject image based on an image signal acquired by imaging. This is a method of generating an edge to be searched from the edge and the subject image and calculating the similarity between the model edge and the edge to be searched.

When the user wants to execute the input mode, the user moves the head to direct the camera 105 to the keyboard KB. At that time, the control unit 107 causes the image display unit 104 to display an image as shown in FIG. 6, for example, so that the user who has observed this image has a part of the keyboard KB within the angle of view of the camera 105. I understand that.

The control unit 107 waits until at least a part of the keyboard KB is captured by the camera 105 in step S11 of step S5, and as a result of pattern matching, determines that at least a part of the keyboard KB is captured (described above If the similarity exceeds the threshold), the flow proceeds to step S12.

At step S12, the control unit 107 shifts to the input mode. Along with the execution of the input mode, the control unit 107 can cause the image display unit 104 to display a screen on which the input character can be confirmed as shown in FIG.

When transitioning to the input mode, the control unit 107 can deduce that the imaged subject is the same as the keyboard KB having the shape stored in the memory 108, so that the origin (for example, the upper left corner of the keyboard KB) It is assumed that position coordinates of each key, character, etc. in the two-dimensional coordinate system as a reference become known. Therefore, even if the key KY is covered with the finger (or a mere finger) FG on which the seal mark MK is attached, characters and the like formed thereon can be recognized.

In the subsequent step S13, the control unit 107 searches by pattern matching whether the seal mark MK (or the finger FG) is present in the subject imaged by the camera 105. If it is determined that the seal mark MK (or the finger FG) is present, the control unit 107 subsequently detects that the finger (or a mere finger) FG to which the seal mark MK has been attached performs a key operation by the following detection method To judge. The detection methods (1) to (3) use pattern matching.
(1) The finger (or mere finger) FG on which the seal mark MK is attached is retained on any key KY of the keyboard KB for a predetermined time (for example, 1 second) or more.
(2) The finger (or mere finger) FG on which the seal mark MK is attached is displaced in the direction in which any key KY of the keyboard KB is pressed. This can be read by the displacement of the edge of the key KY.
(3) Displaced in the direction in which one of the keys KY of the keyboard KB overlapping the finger (or mere finger) FG on which the seal mark MK is attached is pressed. However, “overlapping key” refers to a key on which the centers of the seal mark MK or the finger FG overlap.
(4) The microphone 106 collects a keying sound emitted when the key KY of the keyboard KB is pressed.

Two or more of the above detection methods (1) to (4) may be used in combination. In particular, the detection accuracy is improved by using the detection method (4) in combination with any of the detection methods (1) to (3).

Here, when the user performs a key operation, the control unit 107 determines that the finger (or a mere finger) FG on which the seal mark MK is attached has performed the key operation through the above detection method, and the subsequent step S14. Then, the character or the like of the key KY operated by the key operation is determined, displayed as an input character or the like on the screen shown in FIG. 6, and stored in the memory 108, and the flow proceeds to step S15. On the other hand, when it is not determined that the finger (or a mere finger) FG to which the seal mark MK is attached has performed the key operation, the control unit 107 shifts the flow to step S15.

In step S15, when the control unit 107 determines that at least a part of the keyboard KB is not captured by the camera 105, the control unit 107 recognizes that the user wants to end the input mode, and shifts the flow to step S16. In step S16, the control unit 107 determines, for example, a character or the like (characters or the like displayed on the screen of FIG. 6 in the order of being input) which have been input and stored so far. This enables the HMD 100 to realize more sophisticated operations. However, if there is not even one character input, the control unit 107 treats that there is no input.

On the other hand, when at least a part of the keyboard KB is still imaged by the camera 105, the control unit 107 infers that the user has a character or the like to be added in step S15, and returns the flow to step S13. Perform the same input operation. As a result, it becomes possible to input a plurality of characters etc. For example, in the case of using for password input, the password becomes complicated and security becomes stronger.

According to the present embodiment described above, the same camera is used to capture an image of the keyboard KB as an input member and the finger FG performing key operation as an input pointer, and the user actually uses the finger FG. Since the input can be performed by touching the key KY to be input, an intuitive input operation is possible with a simple configuration. The relationship between the finger and the key in contact with each other hardly affects the parallax due to the difference between the line of sight of the user and the optical axis of the camera.

FIG. 7 is a view showing an input member according to a modification. The input member shown in FIG. 7 may be, for example, a controller CT of a working machine. The controller CT has three keys meaning "up", "down" and "stop", and does not connect to the HMD 100, but with a finger (or a finger) to which the user applies the seal mark MK. By performing the operation, a working machine (not shown) can be operated. Specifically, when the control unit 107 determines that the controller CT has been imaged by the camera 105 by storing the shape of the controller CT in advance in the memory 108, the work machine can be executed by executing the above-described input mode. Can work. Not only the standardized keyboard as described above, but also the shape of an arbitrary controller or the like as in the present modification may be registered (stored) and used as an input member.

The same input can be performed by capturing an image of a keyboard or the like using a camera mounted on a tablet or the like. Further, although the user's finger is used as the input indicator, the key may be indicated using a tip of a pen or a bar whose shape is stored in advance.

The present invention is not limited to the embodiments and modifications described in the specification, and including other embodiments and modifications from the embodiments, modifications and technical ideas described in the present specification. It will be apparent to those skilled in the art. The description and the embodiments of the specification are for the purpose of illustration only, and the scope of the present invention is indicated by the following claims.

100 HMD
101 frame 102 rectangular frame 103 main unit 104 image display unit 105 camera 106 microphone 107 control unit 108 memory CB control box CD wiring CT controller FG finger KB keyboard KY key MK seal mark

Claims

An input attached to the head of the user, forming at least two or more types of characters, numbers and / or symbols in the division area, and indicating the division area with an input indicator used by the user A wearable terminal for inputting information through members,
A display unit that displays an image so that the user can view it;
A camera which picks up an object and outputs an image signal;
A processing unit that processes an image signal output from the camera;
A storage unit for storing features of an image of the input member;
The processing unit is configured such that a subject captured by the camera is at least a part of the input member based on an image signal of the subject output from the camera and a feature of the image of the input member stored in the storage unit. When it is determined that the input mode is started, execution of the input mode is started, and when it is determined that the input indicator has instructed any of the divided areas of the input member while the input mode is being executed, A wearable terminal for inputting information corresponding to the characters, numbers and / or symbols formed in the designated division area.
The wearable terminal according to claim 1, wherein the input member is a keyboard or an object shaped like a keyboard.
The wearable terminal according to claim 2, wherein the partition area is a key of the keyboard or an object shaped in the keyboard.
The indicator is provided with a unique mark, the storage unit stores the features of the unique mark, and the processing unit stores the image signal of the subject output from the camera and the storage unit. When it is determined that the input indicator with the unique mark indicates any of the divided areas of the input member based on the stored characteristic of the unique mark, the instructed divided area is determined The wearable terminal according to any one of claims 1 to 3, wherein information corresponding to the formed letters, numbers and / or symbols is input.
The indicator has a unique shape, the storage unit stores features of the unique shape of the pointer, and the processing unit generates an image signal of an object output from the camera, and When it is determined that the input indicator having the unique shape has instructed any of the divided regions of the input member based on the characteristic of the unique shape of the indicator stored in the storage unit The wearable terminal according to any one of claims 1 to 3, wherein information corresponding to the characters, numbers and / or symbols formed in the divided area is input.
The wearable terminal according to claim 5 or 6, wherein the indicator is a finger.
The processing unit, when detecting that the input indicator has stayed in any of the divided areas of the input member for a predetermined time or more, determines that the divided area is instructed. Wearable terminal as described.
The processing unit, when detecting that the input indicator overlapping with any of the divided regions of the input member is displaced in a predetermined direction, determines that the divided region is instructed. Wearable terminal as described.
The processing unit according to any one of claims 1 to 8, wherein the processing unit determines that the divided area is instructed when detecting that the divided area of the input member overlapping with the input indicator is displaced in a predetermined direction. Wearable terminal.
It has sound acquisition means for acquiring a unique sound generated when the section area of the input member is displaced, the storage unit stores information on the unique sound, and the processing unit The apparatus according to any one of claims 1 to 9, wherein, when the sound acquisition unit detects the unique sound in a state where the input indicator overlaps any of the divided areas of the input member, it is determined that the divided area is indicated. The wearable terminal described in.