KR102148648B1 - Wearable mobile and method using object for pointer - Google Patents

Abstract

The present invention relates to a glasses-type mobile terminal and a control method thereof, and more particularly, to a glasses-type mobile terminal capable of recognizing an arbitrary object as a pointer and controlling motion through movement of the pointer, and a control method thereof. According to an embodiment of the present invention, the mobile terminal determines at least one preliminary pointer in the display unit, the camera, and the image captured through the camera, and determines any one of the preliminary pointers as the first pointer. It may include a control unit.

Description

Glasses-type mobile terminal using an object as a pointer and its control method {WEARABLE MOBILE AND METHOD USING OBJECT FOR POINTER}

The present invention relates to a glasses-type mobile terminal and a control method thereof, and more particularly, to a glasses-type mobile terminal capable of recognizing an arbitrary object as a pointer and controlling motion through movement of the pointer, and a control method thereof.

Terminals can be divided into mobile/portable terminals and stationary terminals depending on whether they can be moved. Again, the mobile terminal can be divided into a handheld terminal and a vehicle mount terminal according to whether the user can directly carry it.

As the functions are diversified, such a terminal is in the form of a multimedia player with complex functions such as, for example, taking photos or videos, playing music or video files, and receiving games and broadcasts. It is being implemented.

In order to support and increase the function of the terminal, it may be considered to improve the structural part and/or the software part of the terminal.

In recent years, a user can simultaneously receive an object in a field of view and information corresponding thereto without directly manipulating the mobile terminal through a glasses-type mobile terminal. However, it is difficult for a user to accurately determine a desired object or selection area through a glasses-type mobile terminal.

An object of the present invention is to provide a glasses-type mobile terminal capable of recognizing an arbitrary object as a pointer and controlling motion through movement of the pointer, and a control method thereof.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. I will be able to.

A glasses-type mobile terminal according to an embodiment of the present invention for realizing the above object includes a display unit; camera; And a control unit determining at least one preliminary pointer in the image captured by the camera, and determining any one of the preliminary pointers as a first pointer, wherein the control unit is configured to correspond to the movement of the first pointer. Function can be performed.

In addition, a control method of a glasses-type mobile terminal according to an embodiment of the present invention for realizing the above-described problem includes: determining at least one preliminary pointer in an image captured by the camera; Determining any one of the reserved pointers as a first pointer; And performing a function corresponding to the movement of the first pointer.

By using the glasses-type mobile terminal according to at least one embodiment of the present invention configured as described above, a user can easily use an arbitrary object as a pointer.

In particular, according to the present invention, an object or information area desired by a user can be conveniently selected by using an object determined as a pointer.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.
2 shows an example of a glasses-type mobile terminal 200 as an embodiment to which the present invention is applied.
3 is a flowchart illustrating an example of a method of determining an arbitrary object as a pointer as an embodiment to which the present invention is applied.
4 is a diagram illustrating an example of determining a spare pointer in an embodiment to which the present invention is applied.
FIG. 5 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.
6 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.
7 is a diagram illustrating an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.
FIG. 8 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.
9 illustrates an example of determining a plurality of objects as pointers among objects determined as preliminary pointers as an embodiment to which the present invention is applied.
10 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.
11 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.
12 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.
13 illustrates an example of determining a selection area by using a pointer as an embodiment to which the present invention is applied.
14 is a diagram illustrating an example of determining a selection area using a plurality of pointers as an embodiment to which the present invention is applied.

Hereinafter, a mobile terminal related to the present invention will be described in more detail with reference to the drawings. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves.

Mobile terminals described herein include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, glasses type terminals. Etc. may be included. However, it will be readily apparent to those skilled in the art that the configuration according to the embodiment described in the present specification may also be applied to a fixed terminal such as a digital TV or a desktop computer, except when applicable only to a mobile terminal.

Hereinafter, a mobile terminal 100 to which the present invention is applied will be described with reference to the drawings.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an audio/video (A/V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, and an interface. The unit 170, the control unit 180, and the power supply unit 190 may be included. Since the components shown in FIG. 1 are not essential, a mobile terminal having more components or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the mobile terminal 100 and a wireless communication system or between the mobile terminal 100 and a network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The broadcast reception module 111 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching of at least two broadcast channels.

The broadcast management server may mean a server that generates and transmits a broadcast signal and/or broadcast-related information, or a server that receives and transmits a previously-generated broadcast signal and/or broadcast-related information to a terminal. The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but also a broadcast signal in a form in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcast-related information means information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast-related information may exist in various forms. For example, it may exist in the form of an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or an Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H).

The broadcast receiving module 111 includes, for example, Digital Multimedia Broadcasting-Terrestrial (DMB-T), Digital Multimedia Broadcasting-Satellite (DMB-S), Media Forward Link Only (MediaFLO), and Digital Video Broadcasting (DVB-H). -Handheld), DVB-CBMS (Convergence of Broadcasting and Mobile Service), OMA-BCAST (Open Mobile Alliance-BroadCAST), CMMB (China Multimedia Mobile Broadcasting), MBBMS (Mobile Broadcasting Business Management System), ISDB-T (Integrated Services) Digital broadcasting signals can be received using a digital broadcasting system such as Digital Broadcast-Terrestrial). Of course, the broadcast reception module 111 may be configured to be suitable for not only the digital broadcasting system described above, but also other broadcasting systems.

The broadcast signal and/or broadcast related information received through the broadcast reception module 111 may be stored in the memory 160.

The mobile communication module 112 includes at least one of a base station, an external terminal, and a server on a mobile communication network such as Global System for Mobile communications (GSM), Code Division Multiple Access (CDMA), and Wideband CDMA (WCDMA) (not limited thereto). It transmits and receives radio signals with Hana. The wireless signal may include an audio call signal, a video call signal, or various types of data according to transmission and reception of text/multimedia messages.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built-in or external to the mobile terminal 100. Wireless Internet technologies include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), GSM, CDMA, WCDMA, LTE (Long Term). Evolution) (but not limited to), and the like may be used.

From the point of view that wireless Internet access through Wibro, HSDPA, GSM, CDMA, WCDMA, LTE, etc. is made through a mobile communication network, the wireless Internet module 113 performing wireless Internet access through the mobile communication network is the mobile communication module. It may be understood as a kind of (112).

The short-range communication module 114 refers to a module for short-range communication. As short range communication technology, Bluetooth, RFID (Radio Frequency Identification), Neighbor Magnetic Field Communication (NFC), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), ZigBee, etc. can be used. have.

The location information module 115 is a module for obtaining a location of a mobile terminal, and a representative example thereof is a Global Position System (GPS) module. According to the current technology, the GPS module 115 calculates distance information and accurate time information away from three or more satellites, and then applies a trigonometric method to the calculated information, thereby providing a three-dimensional string according to latitude, longitude, and altitude. Location information can be accurately calculated. Currently, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is widely used. In addition, the GPS module 115 may calculate speed information by continuously calculating the current location in real time.

Referring to FIG. 1, an audio/video (A/V) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes an image frame such as a still image or a video obtained by an image sensor in a video call mode or a photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. Two or more cameras 121 may be provided depending on the use environment.

The microphone 122 receives an external sound signal by a microphone in a call mode, a recording mode, an sound recognition mode, and the like and processes it as electrical sound data. In the case of a call mode, the processed sound data may be converted into a format that can be transmitted to a mobile communication base station through the mobile communication module 112 and then output. Various noise removal algorithms may be implemented in the microphone 122 to remove noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data for the user to control the operation of the terminal. The user input unit 130 may be composed of a button 136 positioned on the front, rear, or side surfaces of the mobile terminal 100, and a touch sensor (positive pressure/power failure) 137, and although not shown, a keypad ), a dome switch, a jog wheel, a jog switch, and the like may be further included. The sensing unit 140 detects the current state of the mobile terminal 100, such as an open/closed state of the mobile terminal 100, the position of the mobile terminal 100, the presence of user contact, the orientation of the mobile terminal, and acceleration/deceleration of the mobile terminal. Thus, a sensing signal for controlling the operation of the mobile terminal 100 is generated. For example, when the mobile terminal 100 is in the form of a slide phone, it is possible to sense whether the slide phone is opened or closed. In addition, it is also possible to sense whether the power supply unit 190 supplies power or whether the interface unit 170 is coupled to an external device. Meanwhile, the sensing unit 140 may include a proximity sensor 141. (The proximity sensor will be described later).

The output unit 150 is for generating output related to visual, auditory, or tactile sense, and includes a display unit 151, an audio output module 152, an alarm unit 153, and a haptic module 154. I can.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, when the mobile terminal is in a call mode, a user interface (UI) or a graphical user interface (GUI) related to a call is displayed. When the mobile terminal 100 is in a video call mode or a photographing mode, a photographed or/and received image, UI, or GUI is displayed.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. display) and a 3D display.

Some of these displays may be configured as a transparent type or a light-transmitting type so that the outside can be seen through them. This may be referred to as a transparent display, and a representative example of the transparent display is TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light transmission type structure. With this structure, the user can see an object located behind the terminal body through an area occupied by the display unit 151 of the terminal body.

Two or more display units 151 may exist depending on the implementation form of the mobile terminal 100. For example, in the mobile terminal 100, a plurality of display units may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces.

When the display unit 151 and the touch sensor 137 form a mutual layer structure or are integrally formed (hereinafter referred to as'touch screen'), the display unit 151 may be used as an input device in addition to an output device. I can. The touch sensor may be stacked on the display unit 151 to form a layer structure, for example, in the case of having a shape of a touch film, a touch sheet, a touch pad, or the like, or integrated into the display unit 151 Can be made.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or a capacitance generated at a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only a touched position and area, but also a pressure at the time of touch.

When there is a touch input to the touch sensor, a signal(s) corresponding thereto is transmitted to a touch controller (not shown). The touch controller processes the signal(s) and then transmits the corresponding data to the controller 180. As a result, the controller 180 can know whether an area of the display unit 151 is touched.

The proximity sensor 141 may be disposed in an inner area of the mobile terminal surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity using the force of an electromagnetic field or infrared rays without mechanical contact. Proximity sensors have a longer lifespan and higher utilization than contact sensors.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, and the like. When the touch screen is capacitive, it is configured to detect the proximity of the pointer by a change in an electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, an action of allowing the pointer to be located on the touch screen without being in contact with the pointer is referred to as "proximity touch", and the The act of actually touching the pointer on the touch screen may be referred to as "contact touch". A position at which a proximity touch is performed with a pointer on the touch screen may mean a position at which the pointer corresponds vertically to the touch screen when the pointer is proximity touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (eg, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, etc.). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a sound recognition mode, a broadcast reception mode, and the like. The sound output module 152 also outputs sound signals related to functions (eg, call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. The sound output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event in the mobile terminal 100. Examples of events occurring in the mobile terminal include call signal reception, message reception, key signal input, and touch input. The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than a video signal or an audio signal, for example, by vibration. Since the video signal or audio signal may also be output through the display unit 151 or the sound output module 152, in this case, the display unit 151 and the sound output module 152 are a type of the alarm unit 153. It can also be classified.

The haptic module 154 generates various tactile effects that a user can feel. A typical example of the tactile effect generated by the haptic module 154 is vibration. The intensity and pattern of the vibration generated by the haptic module 154 can be controlled. For example, different vibrations may be synthesized and output or may be sequentially output.

In addition to vibration, the haptic module 154 is used for stimulation such as an arrangement of pins that move vertically with respect to the contact skin surface, a blowing force or suction force of air through a jet or inlet, a grazing on the skin surface, contact of an electrode, electrostatic force, etc. It can generate various tactile effects, such as the effect by the effect and the effect by reproducing the feeling of cooling and warming using an endothermic or heat generating element.

The haptic module 154 may not only transmit a tactile effect through direct contact, but may also be implemented so that a user can feel the tactile effect through muscle sensations such as a finger or an arm. Two or more haptic modules 154 may be provided depending on the configuration aspect of the mobile terminal 100.

The memory unit 160 may store a program for processing and control of the controller 180, and temporarily store input/output data (eg, a phone book, a message, an audio, a still image, a video, etc.) It can also perform a function for. The memory unit 160 may store a frequency of use for each of the data (eg, a frequency of use for each phone number, each message, and each multimedia).

In addition, the memory unit 160 may store data on vibrations and sounds of various patterns output when a touch input on the touch screen is performed.

The memory 160 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), and RAM. (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk. The mobile terminal 100 may operate in connection with a web storage that performs a storage function of the memory 160 over the Internet.

The interface unit 170 serves as a passage for all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device or receives power and transmits it to each component inside the mobile terminal 100, or transmits data inside the mobile terminal 100 to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port to connect a device equipped with an identification module, an audio input/output (I/O) port, A video input/output (I/O) port, an earphone port, etc. may be included in the interface unit 170.

The identification module is a chip that stores various types of information for authenticating the right to use the mobile terminal 100, and includes a user identification module (UIM), a subscriber identification module (SIM), and a universal user authentication module ( Universal Subscriber Identity Module, USIM), etc. may be included. A device equipped with an identification module (hereinafter,'identification device') may be manufactured in the form of a smart card. Therefore, the identification device can be connected to the terminal 100 through the port.

When the mobile terminal 100 is connected to an external cradle, the interface unit becomes a path through which power from the cradle is supplied to the mobile terminal 100, or various command signals input from the cradle by a user are moved. It can be a passage to the terminal. Various command signals or the power input from the cradle may be operated as signals for recognizing that the mobile terminal is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the mobile terminal. For example, it performs related control and processing for audio calls, data communication, video calls, and the like. The controller 180 may also include a multimedia module 181 for playing multimedia. The multimedia module 181 may be implemented in the controller 180 or may be implemented separately from the controller 180.

The controller 180 may perform a pattern recognition process capable of recognizing a handwriting input or a drawing input performed on the touch screen as characters and images, respectively.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for the operation of each component. The power supply unit 190 may include, for example, a battery, a connection port, a power supply control unit, and a charge monitoring unit.

The battery may be a built-in battery made to be rechargeable, and may be detachably coupled to the terminal body for charging or the like. The connection port may be configured as an example of an interface 170 to which an external charger that supplies power for charging a battery is electrically connected.

Various embodiments described herein may be implemented in a recording medium that can be read by a computer or a similar device using, for example, software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions, in some cases herein. The described embodiments may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in the memory 160 and executed by the controller 180.

2 shows an example of a glasses-type mobile terminal as an embodiment to which the present invention is applied.

The glasses-type mobile terminal 200 may include a glasses leg 223, a central support 224, and a nose support 225. In the example of FIG. 2, the central support 224 is connected to the glasses leg 223. Further, the glasses-type mobile terminal 200 in the form of glasses shown in FIG. 2 may not include a spectacle lens and a lens frame for supporting the spectacle lens.

The computing system 226 is shown to be positioned over the temple 223 and the central support 224 of the spectacled mobile terminal 200. However, the location of the computing system 226 will not be limited thereto. The computing system 226 corresponds to a main body of a general mobile terminal 100 and may include various devices/modules described above with reference to FIG. 1.

Camera 228 is shown in FIG. 2 to be located on the front side of computing system 226. The camera 228 is the same as the camera 121 described above with reference to FIG. 1, but a different identification code is used to emphasize that the camera 228 is a camera positioned on the front side of the glasses-type mobile terminal 200. The camera 228 may be located elsewhere on the glasses-type mobile terminal 200. The camera 228 is configured to capture images having various image quality and frame rates. In particular, the camera 228 may be used to obtain the same image as the field of view viewed by the user from the front portion of the glasses-type mobile terminal 200.

As shown in FIG. 2, the transparent display 230 is in the form of a head mounted display (HMD), and is shown to be positioned on the central support 234. The HMD type means a display type that can be mounted on the head and present an image directly in front of the user's eyes. That is, when the user wears the glasses-type mobile terminal 200, it may be positioned on the central support 224 to face the user's eyes so that an image can be directly provided in front of the user's eyes.

The output image output through the transparent display 230 is superimposed on the field of view (hereinafter referred to as a general field of view) in which the user looks at an object in daily life. That is, the glasses-type mobile terminal 200 may provide an augmented reality (AR) by using the transparent display 230. Augmented reality is a technology that displays a single image by superimposing a 3D virtual image on a real image or background. A technique for implementing such augmented reality is a known technique, and thus a detailed description thereof will be omitted.

The material of the transparent display 230 for overlapping the general field of view and the output image may be divided into a case of transparent and a case of non-transparent.

According to an embodiment of the present invention, the transparent display 230 is provided with a transparent material, so that the transparent image transmitted through the transparent display 230 and the output image output through the transparent display 230 are overlapped by the user. have.

According to another embodiment of the present invention, the glasses-type mobile terminal 200 may include a non-transparent transparent display 230, and an image provided through the transparent display 230 is transmitted through one of the user's two eyes. At the same time as being provided, a view of a general object may be provided through another eye.

According to another embodiment of the present invention, the glasses-type mobile terminal 200 may include a non-transparent transparent display 230, and may provide an augmented reality based on an image received through the camera 228 as a background. have.

The glasses-type mobile terminal 200 according to an embodiment of the present invention may further include an eye tracking sensor (not shown, an eye tracking sensor). The eye tracking sensor refers to a sensor capable of detecting a direction of a user's pupils, a movement of the pupils, and/or a position of the pupils. The computing system 226 may determine where the user's gaze is directed based on a direction in which the pupil is directed, sensed by the eye tracking sensor. The eye tracking sensor may be disposed to face the user's pupils when the glasses-type mobile terminal 200 is worn by the user.

In the example referenced in FIG. 2, the computing system 226 may include user input units 210, 211 and 212 for receiving input from a user. The user input units 210, 211 and 212 are provided for a control method according to an embodiment of the present invention. The user input units 210, 211, 212 may be formed of a physical button or a touch pad (not limited thereto), and may sense an input selected (touched) by a user using a finger and transmit it to the controller 180 . The user input units 210, 211, 212 may include a drag pad 212. The drag pad 212 is an input module that enables a user to execute a predetermined function of the glasses-type mobile terminal 200 (or other external device connected to the glasses-type mobile terminal 200) based on a touch. The drag pad 212 may be provided on one or both of the spectacle legs 223 composed of two, senses the position or movement of the user's finger, and transmits the result to the computing system 226. For example, the drag pad 212 may detect whether the finger is dragged toward or in the opposite direction of the user's viewing direction, and may transmit the detected result to the computing system 226.

For convenience of explanation, it is assumed that the mobile terminal mentioned below includes at least one of the components shown in FIG. 1. Also, a graphic in the form of an arrow or a finger for pointing to a specific object or selecting a menu on the display unit 151 is usually referred to as a pointer or a cursor. However, in the case of a pointer, it is often used interchangeably to mean a finger or a stylus pen for gesture or touch manipulation. Therefore, in this specification, in order to clearly distinguish the two, a graphic displayed on the display is referred to as a cursor, and a physical means such as a finger or a stylus pen that can perform a touch, a proximity touch, or a gesture is referred to as a pointer. Here, the gesture may be recognized by a proximity sensor or a camera. In the present invention, it is assumed that a gesture through a pointer is recognized through a camera unless otherwise specified.

According to an embodiment of the present invention, there is provided a method of determining an arbitrary object as a pointer and determining an object or a selection area desired by a user by using the determined pointer. Hereinafter, a method of using an object as a pointer using a mobile terminal according to an embodiment of the present invention will be described. In this specification, it is assumed that the mobile terminal is a glasses-type mobile terminal. In particular, it is preferable that the mobile terminal according to the present embodiment is provided with a camera 121 so as to photograph a direction coincident with the user's gaze direction. This is to facilitate the selection of a pointer within a shooting range similar to the user's gaze range and inputting commands through it. Of course, this is for convenience of description, and the present invention is not necessarily limited to a glasses-type mobile terminal.

First, a process of setting an arbitrary object as a pointer will be described with reference to FIG. 3.

3 is a flowchart illustrating a method of setting an object as a pointer according to an embodiment to which the present invention is applied.

When a pointer setting command is input, the controller 180 may display information related to pointer setting (S310). Here, the information related to the pointer setting may include a guide message or visual effect for setting an arbitrary object included in the image captured by the camera as a pointer, a screen shot through the camera 121, and the like. As described above, when the camera 121 is provided to face the user's gaze direction, the captured screen may be a screen in the same direction as the user's gaze. The pointer setting command may be input in the form of key button manipulation, touch input, or voice input.

The controller 180 may determine at least some of the object(s) photographed through the camera 121 as a preliminary pointer (S320). Here, the preliminary pointer may represent an object that can be used as a pointer in the screen photographed by the camera 121. The preliminary pointer may be selected regardless of its shape, size, or color, but it is preferable to select an object having a color that contrasts with the background by a predetermined degree or more, and a shape that is easy to identify. The contrast of colors may be determined by calculating a color code value of a color recognized through the camera 121. For example, a color may be expressed from 000000 to FFFFFF based on RGB through a color code value. Based on this, the controller 180 may determine the contrast of colors by comparing a color code value of an object with a background corresponding to each of red, green, and blue. An example of determining the preliminary pointer by the controller 180 will be described later with reference to FIG. 4.

The controller 180 may set at least one object among the preliminary pointers to be used as a pointer (S330). The pointer setting criterion may include input of a user's selection command and whether or not a predetermined criterion is satisfied. Examples of the predetermined criterion include whether the object is the same as the object previously set as a pointer, and the degree to which the color of the preliminary pointer contrasts with the background. A method of determining a pointer among the preliminary pointers will be described in more detail later with reference to FIGS. 5 to 9.

When the pointer setting is completed, the controller 180 may control the mobile terminal in response to the movement of the pointer set in the image captured by the camera (S340). For example, when a finger is determined as a pointer, the user may select another object with the finger or select an area through a gesture. Accordingly, the controller 180 may output information related to the selected object or the selection region or perform a function related thereto. A method of using the pointer will be described in detail with reference to FIGS. 10 to 14.

Hereinafter, an example of determining a preliminary pointer among objects will be described with reference to FIG. 4.

4 is a diagram illustrating an example of determining a spare pointer in an embodiment to which the present invention is applied.

The controller 180 may display an image captured by the camera 121 on the display unit 151 as shown in (a) of FIG. 4. If the user inputs a pointer setting command, the controller 180 may determine at least one preliminary pointer from an image displayed on the display unit 151. Preferably, the controller 180 may determine, as a preliminary pointer, at least one object having a color contrasting with the background in the form of an elongated bar. For example, the controller 180 may determine a ballpoint pen, a ruler, a left hand with an index finger, and a right hand holding a ballpoint pen as a preliminary pointer as shown in FIG. 4B. In addition, the controller 180 may display a predetermined type of visual effect to the preliminary pointer or adjacent thereto so that the user can identify the object determined as the preliminary pointer. For example, an elliptical visual effect may be displayed around an object determined as a preliminary pointer. In addition, the controller 180 may provide a user with a guide for determining a pointer. As an example of guidance, the controller 180 displays guidance such as "Please select a pointer" or "Move the object to the pointer selection area" on the display unit 151, or transmits a corresponding voice through the sound output unit. Can be printed.

Hereinafter, a method of determining a first object as a pointer among the preliminary pointers will be described with reference to FIGS. 5 to 8. Hereinafter, it is assumed that the first object means an object that can actually become a pointer, and the second object means an object in which the first object is photographed through the camera 121 and displayed on the display unit 151.

As an example of how to determine a pointer, a visual effect is displayed in a specific area of the display, and when a spare pointer enters a specific area (hereinafter referred to as "pointer selection area"), the corresponding spare pointer is set as a pointer. I can. This will be described with reference to FIG. 5.

FIG. 5 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.

The controller 180 may set a first object corresponding to the second object moved to the pointer selection area as a pointer. The user can move the second object by moving the first object. For example, while determining a preliminary pointer as shown in FIG. 4B, the controller 180 displays the second object and the pointer selection area 500 on the display unit 151 as shown in FIG. 5A. I can. If the user moves the right hand 510 holding the first object, the ballpoint pen, as shown in FIG. 5B and adjusts the second object 520 to be stopped in the pointer selection area 500 for a predetermined time or longer, the controller Reference numeral 180 may determine that the user has determined that the right hand 510 holding the ballpoint pen as the first object is determined as a pointer. In addition, the controller 180 may set the right hand 510 holding the ballpoint pen as a pointer.

6 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.

The controller 180 may set the first object moved to the pointer selection area 300 as a pointer. For example, while determining a preliminary pointer as shown in FIG. 4B, the controller 180 may display the pointer selection area 500 on the display unit 151 as shown in FIG. 6A. In FIG. 6, a portion displayed on the display unit 151 may be displayed in a translucent manner, so that a user can recognize an object in a portion where the display unit 151 is displayed or a portion where the display unit 151 is displayed. . If the user controls the right hand 510 holding the first object, the ballpoint pen, to stop in the pointer setting area 500 for a predetermined time or longer, the controller 180 controls the right hand 510 holding the first object, the ballpoint pen, It is determined that is determined as a pointer, and a user input can be set to be performed through the tip of the ballpoint pen.

The method of determining the pointer of FIG. 5 and the method of determining the pointer of FIG. 6 are similar in that the user moves the first object to be used as a pointer. However, in the method of determining the pointer of FIG. 6, the second object is not displayed on the display unit 151, and the first object to be used as a pointer is moved to correspond to the pointer setting area 500 in the display unit 151. However, the method of determining the pointer of FIG. 5 differs in that the second object is displayed on the display unit 151 and the second object is moved to the pointer setting area 500.

7 is a diagram illustrating an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.

When the user moves the first object in the pointer selection area 500 so that the second object has a predetermined movement, the controller 180 may set the first object as a pointer. Here, the predetermined movement is a specific movement for determining an object determined as a preliminary pointer as a pointer, and may include various movements such as regularly moving left and right, and regularly moving vertically. For example, the controller 180 may determine a preliminary pointer among objects and display it as shown in FIG. 7A. If the user regularly shakes the hand 510 holding the ballpoint pen, which is one of the first objects determined as a preliminary pointer, as shown in FIG. 7B, the controller 180 may perform a second object corresponding to the first object. The object 520 may be displayed on the display unit 151 so that it regularly moves left and right within the pointer selection area 500. In addition, when it is determined that the movement of the second object 520 is a predetermined movement, the controller 180 determines that the right hand 510 holding the ballpoint pen corresponding to the second object 520 is determined as a pointer. , User input can be set to be performed through the tip of a ballpoint pen.

FIG. 8 illustrates an example of determining one of objects determined as a preliminary pointer as a pointer as an embodiment to which the present invention is applied.

The controller 180 may set the moved first object as a pointer so that the user has a predetermined movement in a certain area. For example, the controller 180 may display only a guide for determining a pointer as shown in FIG. 8A on the display unit 151, and a screen photographed by the camera 121 may not be displayed. In this case, when the user regularly shakes the first object 510 left and right, the controller 180 may detect the movement of the first object 510 by the camera 121. If the controller 180 determines that the movement of the first object 510 is a predetermined movement, it is determined that the right hand 510 holding the ballpoint pen, which is the first object, is determined as a pointer, and user input through the tip of the ballpoint pen Can be set to do this.

Unlike the method of FIG. 7, the method of FIG. 8 does not display the second object 520 on the display unit 151. In addition, even if the second object does not move within the pointer selection area 500 and the first object 510 moves in an arbitrary area, the object may be set as a pointer.

Also, the controller 180 may set an object that has been used as a pointer in the past as a pointer. For example, the controller 180 may store the shape and color of an object that has been used as a pointer in the past. And, when the user determines the pointer, the controller 180 compares the object detected by the camera 121 with the object used in the past, and indicates that the object detected by the camera 121 is an object that has been used in the past. So you can entice the user to make a choice. Alternatively, you can immediately set an object used in the past as a pointer.

Even if one pointer is set, a user can additionally set a pointer to use a plurality of pointers. Hereinafter, a method of determining a plurality of objects as pointers will be described with reference to FIG. 9.

9 illustrates an example of determining a plurality of objects as pointers among objects determined as preliminary pointers as an embodiment to which the present invention is applied.

The controller 180 may determine one object among the objects determined as the preliminary pointer as the first pointer, and determine another object as the second pointer. For example, the controller 180 may display a pointer selection area 500 and a second object determined as a preliminary pointer on the display unit 151 as shown in FIG. 9A. In addition, the first pointer may be determined by the method described with reference to FIGS. 5 to 8. For example, when the second object 520 corresponding to the right hand holding the ballpoint pen is moved to the pointer selection area 500 as shown in (b) of FIG. 9, the control unit 180 removes the right hand 510 holding the ballpoint pen. Can be determined with 1 pointer. In addition, after the first pointer is determined, the controller 180 may additionally detect the user's pointer selection and determine the second pointer. If the user moves the left hand 910 as shown in FIG. 9(c) and the second object 920 corresponding to the left hand is moved to the pointer selection area 500, the left hand 910 becomes the second pointer. You can decide. In addition, when the user gives a command to end pointer selection, the controller 180 may set the right hand 510 and the left hand 910 holding the ballpoint pen to perform user input as pointers.

The user can accurately select the desired object or information by using the pointer determined by the above-described method. When the user selects an object or a selection area using a pointer, the controller 180 can control the display unit 151 to display information (including text, photos, videos, etc.) corresponding to the selected object or selection area. have.

Hereinafter, a method of using a pointer by a user will be described with reference to FIGS. 10 to 14.

10 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.

The controller 180 may detect movement of the pointer, determine the selection area 900 according to the movement of the pointer, and provide information about text, pictures, tables, etc. in the selection area 900 to the user. Here, the selection area 900 may represent an area selected by the user with a pointer to obtain information.

When the user stops the pointer at a specific point for a predetermined time or more and determines the start point and the end point, the controller 180 may determine the selection area 900 including a picture or character on a line connecting the start point and the end point. . For example, when the user's right hand holding the ballpoint pen is determined as the pointer 510 as shown in FIG. 10A and the pointer 510 is stopped at the lower left part of the'library' for a predetermined time or longer, the controller 180 ) May determine the point at which the pointer 510 stops as the start point of the selection area 900. In addition, when the user stops the pointer 510 at the lower right part of the'library' for a predetermined time or longer as shown in (b) of FIG. 10, the controller 180 selects the point at which the pointer 510 stops. ) Can be determined by the end point. The controller 180 may determine a selection area 900 including a'library', which is a character positioned at an upper end of a line connecting the start point and the end point. In addition, the controller 180 may display information related to a “library” and a library in the area on the display unit 151.

11 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.

When the user determines the start point and the end point through a predetermined motion of the pointer at a specific point, the controller 180 may determine the selection area 900 including a picture or text on a line connecting the start point and the end point. . The predetermined operation may represent an operation such as drawing a figure or text through a pointer, and the predetermined operation may include various embodiments. For example, when the user draws a circle through a pointer in the lower left part of the'library' as shown in FIG. 11(a), the controller 180 selects the center of the circle drawn by the pointer in the selection area 900 Can be determined as the starting point of. In addition, when the user draws a circle with a pointer at the lower right part of the'library' as shown in (b) of FIG. 11, the controller 180 selects the center of the circle drawn by the pointer to the end point of the selection area 900 Can be determined by The controller 180 may determine a selection area 900 including a'library', which is a character positioned at an upper end of a line connecting the start point and the end point. In addition, the controller 180 may display information related to a “library” and a library in the area on the display unit 151.

12 is a diagram illustrating an example of determining a selection area using a pointer as an embodiment to which the present invention is applied.

The controller 180 may determine an area within the single closed loop drawn by the user through the pointer as the selection area 1200. Here, the single closed loop represents a shape with the same starting point and ending point of a straight line or curve, such as a polygon, a circle, or an ellipse. For example, when the user moves the pointer clockwise to draw a closed loop in the form of a rectangle as shown in FIG. 12, the controller 180 selects a selection area 1200 including a'library' and a'park' inside the rectangle. You can decide. In addition, the controller 180 may display information related to the'library' and the'park' in the selection area 1200 on the display unit 151.

13 illustrates an example of determining a selection area by using a pointer as an embodiment to which the present invention is applied.

The controller 180 may determine an area including a line drawn by a user through a pointer as the selection area 1300. For example, when the user moves the pointer and draws a diagonal line as shown in FIG. 13, the controller 180 may determine a rectangle in which the start and end points of the diagonal line are vertices as the selection area 1300. In addition, the controller 180 may display information related to the'library' and the'park' in the selection area 1300 on the display unit 151.

14 is a diagram illustrating an example of determining a selection area using a plurality of pointers as an embodiment to which the present invention is applied.

When the plurality of pointers are stopped at a specific point for a predetermined time or longer, the controller 180 may determine an area including the specific point as the selection area 1400. For example, as shown in FIG. 14, when the right hand 510 and the left hand 910 holding the ballpoint pen are stopped at a specific point for a predetermined time or longer, the control unit 180 controls the right hand 510 and the left hand 910 holding the ballpoint pen. A rectangle including a specific point indicated by this may be determined as the selection area 1400.

In addition, according to an embodiment of the present invention, the above-described method can be implemented as code that can be read by a processor in a medium on which a program is recorded. Examples of media that can be read by a processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and are implemented in the form of carrier waves (for example, transmission over the Internet). Include.

The mobile terminal and its control method described above are not limitedly applicable to the configuration and method of the above-described embodiments, but the embodiments are all or part of each embodiment selectively so that various modifications can be made. It may be configured in combination.

Claims (20)

A display unit;
camera; And
A controller configured to determine at least one preliminary pointer in the image captured by the camera, determine any one of the preliminary pointers as the first pointer, and then perform a function corresponding to the movement of the first pointer,
The control unit,
A glasses-type mobile terminal that performs a function corresponding to a movement of the second pointer after determining another one of the at least one preliminary pointer as a second pointer. delete delete delete delete The method of claim 1,
The control unit,
Determining a first point and a second point according to the movement of the first pointer,
A glasses-type mobile terminal configured to control a second visual effect corresponding to a second area including the first point and the second point to be displayed on the display unit. The method of claim 6,
The second visual effect,
A glasses-type mobile terminal comprising a visual effect for displaying a border of the second region and a visual effect for displaying a character of information corresponding to the second region. The method of claim 6,
The control unit,
A point at which the first pointer stops for a predetermined time is determined as the first point,
A glasses-type mobile terminal configured to determine a point at which the first pointer stops for a predetermined time after the first point is determined as the second point. The method of claim 6,
The control unit,
A point at which the first pointer is moved by a specific movement is determined as the first point,
After the first point is determined, a point at which the first pointer is moved by a specific movement is determined as the second point. The method of claim 1,
The control unit,
A point at which the first pointer stops for a predetermined time is determined as the first point,
A point at which the second pointer stops for a predetermined time is determined as the second point,
Displaying a second area including the first point and the second point on the display unit. delete delete delete delete delete delete delete delete delete delete

Images