KR20100102252A - Mobile terminal and method for inputting instructions thereto - Google Patents

Abstract

PURPOSE: A mobile terminal and a method for inputting instructions thereof are provided to find a location of a central object on a background image or predict motion when the central object is not completely recognized, thereby reliably inputting instructions. CONSTITUTION: A camera(121) consecutively photographs image frames including a background image and a central object in a fixed photographing range. A control unit(180) compares each image frame from a fixed reference image frame with a previous frame through the image frames in an image recognition input mode. The control unit generates one virtual background image by combination of a new background image to connect the new background image with the previous background image.

Description

MOBILE TERMINAL AND METHOD FOR INPUTTING INSTRUCTIONS THERETO}

The present invention relates to a mobile terminal having a camera and a command input method thereof.

Terminal can move And can be divided into a mobile / portable terminal and a stationary terminal depending on whether the mobile terminal is a mobile terminal or a mobile terminal. The mobile terminal may be further classified into a handheld terminal and a vehicle mount terminal according to whether a user can directly carry it.

Such a terminal has various functions, for example, in the form of a multimedia device having multiple functions such as photographing and photographing of a moving picture, reproduction of a music or video file, reception of a game and broadcasting, etc. .

In order to support and increase the function of such a terminal, it may be considered to improve the structural part and / or the software part of the terminal. Accordingly, the terminal is equipped with an increasingly complex user input unit capable of performing all the various functions.

A representative example of a conventional interface operation method for a mobile terminal is a key-based method. Under certain circumstances, the assigned signal is transmitted and interpreted by pressing a specific key to perform a given task. It is still used as a key interface operation method due to its high accuracy. However, only a limited action of pressing a key can be applied, and research into new control methods for communicating with the device has been conducted, and as a result, applications have been applied in many areas.

Touch-based interface control is a more advanced method in key-based techniques. The key-based control technique shows a more organic connection between the user and the device by inputting as it is seen on the touch-based interface while the information on the screen and the input location are different. However, this method also has a limitation in that it must inherit physical limitations as it inherits the limitations of the key input method.

Another method is using a sensing module, for example, an acceleration sensor, which is a technique that is currently widely used. Information on the movement of the terminal can be obtained at a low cost, and is utilized through various applications using the same. However, when actually looking at the terminal employing it, the accelerometer is meaningful when combined with other technologies rather than the main role itself, unlike other interface control technology. In the end, its scope of use is limited.

On the other hand, technology using cameras has attracted much attention. The information contained in the image taken by the camera has more information than that through other input means. That is, the information such as brightness, color, shape, distance, and object tracking obtained through the camera may be intuitive and various information, unlike the limited information obtained from other interface control techniques.

The present invention is to provide a mobile terminal and a command input method thereof that can more conveniently input user commands in operating various functions of the mobile terminal.

In order to solve the above technical problem, the present specification discloses a mobile terminal and various command recognition methods.

A mobile terminal according to an aspect of the present invention uses a plurality of image frames in a camera and image recognition input mode to continuously photograph a plurality of image frames including a background image and a center object within a predetermined shooting range. Each image frame is compared with the previous frame in order starting from the reference image frame of, and a new background image, which is not present in the background image of the previous frame, is cumulatively synthesized so as to be connected with the previous background image. It includes a control unit for controlling to generate.

Preferably, the controller extracts at least one of a position and a displacement of the center object with respect to the generated virtual background image, recognizes it as a user input, and controls to perform a function corresponding to the recognized user input. have.

According to an aspect of the present invention, there is provided a method of inputting a command of a mobile terminal, comprising: extracting a background image of an Nth image frame to generate a virtual background image; and extracting a background image of an N + αth image frame to extract the Nth image frame. And comparing the background image to the background image of the N + α-th image frame and adding a new portion not included in the N-th image frame to the virtual background image.

The mobile terminal according to the at least one embodiment of the present invention configured as described above uses the virtual background image generated cumulatively, even though the central object is not completely recognized in the image photographed by the camera. By knowing the image or predicting the movement, more reliable command input is possible.

Hereinafter, a mobile terminal according to the present invention will be described in more detail with reference to the accompanying drawings. The suffixes "module" and "unit" for components used in the following description are given or used in consideration of ease of specification, and do not have distinct meanings or roles from each other.

The mobile terminal described herein may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), navigation, and the like.

[Description of Fig. 1-Overall Configuration]

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, and an interface. The unit 170, the controller 180, and the power supply unit 190 may be included. The components shown in FIG. 1 are not essential, so that a mobile terminal having more or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and a network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short range communication module 114, and a location information module 115 .

The broadcast receiving module 111 receives a broadcast signal and / or broadcast related information from an external broadcast management server through a broadcast channel.

The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may mean a server that generates and transmits a broadcast signal and / or broadcast related information or a server that receives a previously generated broadcast signal and / or broadcast related information and transmits the same to a terminal. The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but also a broadcast signal having a data broadcast signal combined with a TV broadcast signal or a radio broadcast signal.

The broadcast associated information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast related information may exist in various forms. For example, it may exist in the form of Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H).

The broadcast receiving module 111 may include, for example, Digital Multimedia Broadcasting-Terrestrial (DMB-T), Digital Multimedia Broadcasting-Satellite (DMB-S), Media Forward Link Only (MediaFLO), and Digital Video Broadcast (DVB-H). Digital broadcast signals can be received using digital broadcasting systems such as Handheld and Integrated Services Digital Broadcast-Terrestrial (ISDB-T). Of course, the broadcast receiving module 111 may be configured to be suitable for not only the above-described digital broadcasting system but also other broadcasting systems.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 112 transmits and receives a wireless signal with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The wireless internet module 113 refers to a module for wireless internet access and may be embedded or external to the mobile terminal 100. Wireless Internet technologies may include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like.

The short range communication module 114 refers to a module for short range communication. As a short range communication technology, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like may be used.

The location information module 115 is a module for obtaining a location of a mobile terminal, and a representative example thereof is a GPS (Global Position System) module.

Referring to FIG. 1, the A / V input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video call mode or the photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. Two or more cameras 121 may be provided according to the use environment.

The microphone 122 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, etc., and processes the external sound signal into electrical voice data. The processed voice data may be converted into a form transmittable to the mobile communication base station through the mobile communication module 112 and output in the call mode. The microphone 122 may implement various noise removing algorithms for removing noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data for the user to control the operation of the terminal. The user input unit 130 may include a key pad dome switch, a touch pad (static pressure / capacitance), a jog wheel, a jog switch, and the like.

The sensing unit 140 detects a current state of the mobile terminal 100 such as an open / closed state of the mobile terminal 100, a location of the mobile terminal 100, presence or absence of a user contact, orientation of the mobile terminal, acceleration / deceleration of the mobile terminal, and the like. To generate a sensing signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is in the form of a slide phone, it may sense whether the slide phone is opened or closed. In addition, whether the power supply unit 190 is supplied with power, whether the interface unit 170 is coupled to the external device may be sensed. The sensing unit 140 may include a proximity sensor 141.

The output unit 150 is used to generate an output related to visual, auditory or tactile senses, which includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, and a projector module ( 155) may be included.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, when the mobile terminal is in a call mode, the mobile terminal displays a user interface (UI) or a graphic user interface (GUI) related to the call. When the mobile terminal 100 is in a video call mode or a photographing mode, the mobile terminal 100 displays photographed and / or received images, a UI, and a GUI.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). and at least one of a 3D display.

Some of these displays can be configured to be transparent or light transmissive so that they can be seen from the outside. This may be referred to as a transparent display. A representative example of the transparent display is TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light transmissive structure. With this structure, the user can see the object located behind the terminal body through the area occupied by the display unit 151 of the terminal body.

There may be two or more display units 151 according to the implementation form of the mobile terminal 100. For example, in the mobile terminal 100, a plurality of display portions may be spaced apart from one another, or may be disposed integrally with one another, and may be disposed on different surfaces, respectively.

When the display unit 151 and a sensor for detecting a touch operation (hereinafter, referred to as a touch sensor) form a mutual layer structure (hereinafter referred to as a touch screen), the display unit 151 may be configured in addition to an output device. Can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, or a touch pad.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only the position and area of the touch but also the pressure at the touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits the corresponding data to the controller 180. As a result, the controller 180 can know which area of the display unit 151 is touched.

The proximity sensor 141 may be disposed in an inner region of the mobile terminal surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity without using a mechanical contact by using an electromagnetic force or infrared rays. Proximity sensors have a longer life and higher utilization than touch sensors.

Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the touch screen is configured to detect the proximity of the pointer by the change of the electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of allowing the pointer to be recognized without being in contact with the touch screen so that the pointer is located on the touch screen is referred to as a "proximity touch", and the touch The act of actually touching the pointer on the screen is called "contact touch." The position where the proximity touch is performed by the pointer on the touch screen refers to a position where the pointer is perpendicular to the touch screen when the pointer is in proximity proximity.

The proximity sensor detects a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, and a proximity touch movement state). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output module 152 may also output a sound signal related to a function (eg, a call signal reception sound, a message reception sound, etc.) performed in the mobile terminal 100. The sound output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying occurrence of an event of the mobile terminal 100. Examples of events occurring in the mobile terminal include call signal reception, message reception, key signal input, and touch input. The alarm unit 153 may output a signal for notifying occurrence of an event in a form other than a video signal or an audio signal, for example, vibration. The video signal or the audio signal may be output through the display unit 151 or the audio output module 152, so that they 151 and 152 may be classified as part of the alarm unit 153.

The haptic module 154 generates various tactile effects that the user can feel. Vibration is a representative example of the haptic effect generated by the haptic module 154. The intensity and pattern of vibration generated by the haptic module 154 can be controlled. For example, different vibrations may be synthesized and output or may be sequentially output.

In addition to vibration, the haptic module 154 may be configured to provide a pin array that vertically moves with respect to the contact skin surface, a jetting force or suction force of air through the jetting or suction port, grazing to the skin surface, contact of the electrode, electrostatic force, and the like. Various tactile effects can be generated, such as effects by the endothermic and the reproduction of a sense of cold using the elements capable of endotherm or heat generation.

The haptic module 154 may not only deliver the haptic effect through direct contact, but also may implement the user to feel the haptic effect through a muscle sense such as a finger or an arm. Two or more haptic modules 154 may be provided according to a configuration aspect of the mobile terminal 100.

The projector module 155 is a component for performing an image project function using the mobile terminal 100 and is similar to the image displayed on the display unit 151 in accordance with a control signal of the controller 180 Or at least partly display another image on an external screen or wall.

In detail, the projector module 155 may include a light source (not shown) for generating light (for example, laser light) for outputting the image to the outside, and an image for output to the outside using the light generated by the light source. And an image generating means (not shown), and a lens (not shown) for expanding and outputting the image to the outside at a predetermined focal length. Further, the projector module 155 may include a device (not shown) capable of mechanically moving the lens or the entire module to adjust the image projection direction.

The projector module 155 can be divided into a CRT (Cathode Ray Tube) module, an LCD (Liquid Crystal Display) module and a DLP (Digital Light Processing) module according to the type of the display means. In particular, the DLP module may be advantageous for miniaturization of the projector module 151 by enlarging and projecting an image generated by reflecting light generated from a light source on a DMD (Digital Micromirror Device) chip.

Preferably, the projector module 155 may be provided longitudinally on the side, front or back side of the mobile terminal 100. It goes without saying that the projector module 155 may be provided at any position of the mobile terminal 100 as occasion demands.

The memory unit 160 may store a program for processing and controlling the controller 180, and temporarily stores input / output data (for example, a phone book, a message, an audio, a still image, a video, etc.). It can also perform a function for. The memory unit 160 may also store the frequency of use of each of the data (for example, each telephone number, each message, and frequency of use for each multimedia). In addition, the memory unit 160 may store data regarding vibration and sound of various patterns output when a touch input on the touch screen is performed.

The memory 160 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, It may include a storage medium of at least one type of magnetic disk, optical disk. The mobile terminal 100 may operate in connection with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a path with all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device, receives power, transfers the power to each component inside the mobile terminal 100, or transmits data inside the mobile terminal 100 to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip that stores various types of information for authenticating the usage rights of the mobile terminal 100, and includes a user identification module (UIM), a subscriber identify module (SIM), and a universal user authentication module ( Universal Subscriber Identity Module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device may be connected to the terminal 100 through a port.

The interface unit may be a passage through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, or various command signals input from the cradle by a user may be transferred. It may be a passage that is delivered to the terminal. Various command signals or power input from the cradle may be operated as signals for recognizing that the mobile terminal is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the mobile terminal. For example, perform related control and processing for voice calls, data communications, video calls, and the like. The controller 180 may include a multimedia module 181 for playing multimedia. The multimedia module 181 may be implemented in the controller 180 or may be implemented separately from the controller 180.

The controller 180 may perform a pattern recognition process for recognizing a writing input or a drawing input performed on the touch screen as text and an image, respectively.

The power supply unit 190 receives an external power source and an internal power source under the control of the controller 180 to supply power for operation of each component.

Various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware or a combination thereof.

According to a hardware implementation, embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and the like. It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented in software applications written in a suitable programming language. The software code may be stored in the memory 160 and executed by the controller 180.

[Description on Fig. 2-Instrument Description]

2A is a front perspective view of an example of a mobile terminal or a portable terminal according to the present invention.

The disclosed portable terminal 100 has a terminal body in the form of a bar. However, the present invention is not limited thereto and may be applied to various structures such as a slide type, a folder type, a swing type, a swivel type, and two or more bodies are coupled to be movable relative to each other.

The body includes a casing (casing, housing, cover, etc.) that forms an exterior. In this embodiment, the case may be divided into a front case 101 and a rear case 102. Various electronic components are built in the space formed between the front case 101 and the rear case 102. At least one intermediate case may be further disposed between the front case 101 and the rear case 102.

The cases may be formed by injecting synthetic resin or may be formed of a metal material, for example, a metal material such as stainless steel (STS) or titanium (Ti).

The display unit 151, the audio output unit 152, the camera 121, the user input units 130/131 and 132, the microphone 122, and the interface 170 may be disposed in the terminal body, mainly the front case 101. have.

The display unit 151 occupies most of the main surface of the front case 101. A sound output unit 151 and a camera 121 are disposed in an area adjacent to one end of both ends of the display unit 151 and a user input unit 131 and a microphone 122 are disposed in an area adjacent to the other end. The user input unit 132 and the interface 170 may be disposed on side surfaces of the front case 101 and the rear case 102.

The user input unit 130 is manipulated to receive a command for controlling the operation of the portable terminal 100 and may include a plurality of operation units 131 and 132. The manipulation units 131 and 132 may also be collectively referred to as manipulating portions, and may be employed in any manner as long as the user operates the tactile manner with a tactile feeling.

Content input by the first or second manipulation units 131 and 132 may be variously set. For example, the first operation unit 131 receives commands such as start, end, scroll, and the like, and the second operation unit 132 controls the size of the sound output from the sound output unit 152 or the size of the sound output from the display unit 151 To the touch recognition mode of the touch screen.

FIG. 2B is a rear perspective view of the portable terminal shown in FIG. 2A.

Referring to FIG. 2B, a camera 121 ′ may be additionally mounted on the rear of the terminal body, that is, the rear case 102. The camera 121 'may have a photographing direction substantially opposite to the camera 121 (see FIG. 2A), and may be a camera having different pixels from the camera 121.

For example, the camera 121 has a low pixel so that the user's face is photographed and transmitted to the counterpart in case of a video call, and the camera 121 'photographs a general subject and does not transmit it immediately. It is desirable to have a high pixel because there are many. The cameras 121 and 121 'may be installed in the terminal body so as to be rotatable or pop-upable.

A flash 123 and a mirror 124 are further disposed adjacent to the camera 121 '. The flash 123 shines light toward the subject when the subject is photographed by the camera 121 '. The mirror 124 allows the user to see his / her own face or the like when photographing (self-photographing) the user using the camera 121 '.

The sound output unit 152 'may be further disposed on the rear surface of the terminal body. The sound output unit 152 ′ may implement a stereo function together with the sound output unit 152 (see FIG. 2A), and may be used to implement a speakerphone mode during a call.

The antenna 116 for receiving a broadcast signal may be additionally disposed on the side of the terminal body. The antenna 116, which forms part of the broadcast receiving module 111 (see FIG. 1), can be installed to be able to be drawn out from the terminal body.

A power supply unit 190 for supplying power to the portable terminal 100 is mounted on the terminal body. The power supply unit 190 may be embedded in the terminal body or may be directly detachable from the outside of the terminal body.

The rear case 102 may be further equipped with a touch pad 135 for sensing a touch. Like the display unit 151, the touch pad 135 may also be configured to have a light transmission type. In this case, if the display unit 151 is configured to output visual information from both sides, the visual information may be recognized through the touch pad 135. The information output on both surfaces may be controlled by the touch pad 135. Alternatively, a display may be additionally mounted on the touch pad 135, and a touch screen may be disposed on the rear case 102 as well.

[Description on Fig. 3-Front, Back Touch]

Hereinafter, the operation of the display unit 151 and the touch pad 135 will be described with reference to FIGS. 3A and 3B.

3A and 3B are front views of a portable terminal for explaining an operation state of the mobile terminal according to the present invention.

Various types of time information can be displayed on the display unit 151. [ These pieces of information can be displayed in the form of letters, numbers, symbols, graphics, or icons.

In order to input such information, at least one of the letters, numbers, symbols, graphics, or icons may be displayed in a predetermined arrangement so as to be implemented in the form of a keypad. Such a keypad may be called a 'virtual keypad'.

3A illustrates receiving a touch applied to the virtual keypad through the front of the terminal body.

The display unit 151 may operate in an entire area or may be operated in a divided manner. In the latter case, the plurality of areas can be configured to operate in association with each other.

For example, an output window 151a and an input window 151b are displayed on the upper and lower portions of the display unit 151, respectively. The output window 151a and the input window 151b are areas allocated for outputting or inputting information, respectively. The input window 151b outputs a virtual keypad 151c in which numbers for inputting a telephone number or the like are displayed. When the virtual keypad 151c is touched, numbers and the like corresponding to the touched virtual keypad are displayed on the output window 151a. When the first manipulation unit 131 is operated, a call connection to the telephone number displayed on the output window 151a is attempted.

3B illustrates receiving a touch applied to the virtual keypad through the rear of the terminal body. If FIG. 3A is a portrait in which the terminal body is arranged vertically, FIG. 3B illustrates a landscape in which the terminal body is arranged horizontally. The display unit 151 may be configured to convert the output screen according to the arrangement direction of the terminal body.

3B shows that the text input mode is activated in the mobile terminal. The display unit 151 displays an output window 151a 'and an input window 151b'. A plurality of virtual keypads 151c 'displaying at least one of letters, symbols, and numbers may be arranged in the input window 151b'. The virtual keypads 151c 'may be arranged in the form of a QWERTY key.

When the virtual keypads 151c 'are touched through the touch pad 135 (refer to FIG. 2B), letters, numbers, and symbols corresponding to the touched virtual keypads are displayed on the output window 151a'. As described above, the touch input through the touch pad 135 has an advantage of preventing the virtual keypad 151c 'from being covered by the finger when touched, as compared with the touch input through the display unit 151. When the display unit 151 and the touch pad 135 are transparent, the fingers located at the rear of the terminal body can be visually checked, and thus more accurate touch input is possible.

In addition to the input methods disclosed in the above embodiments, the display unit 151 or the touch pad 135 may be configured to receive a touch input by scrolling. By scrolling the display unit 151 or the touch pad 135, the user may move an object displayed on the display unit 151, for example, a cursor or a pointer located at an icon. Further, when the finger is moved on the display portion 151 or the touch pad 135, the path along which the finger moves may be visually displayed on the display portion 151. [ This may be useful for editing an image displayed on the display unit 151.

One function of the terminal may be executed in response to a case where the display unit 151 (touch screen) and the touch pad 135 are touched together within a predetermined time range. In the case of being touched together, there may be a case where the user clamps the terminal body using the thumb and index finger. For example, the function may include activation or deactivation of the display unit 151 or the touch pad 135.

[Description on Figure 4-Proximity Touch]

The proximity sensor 141 described with reference to FIG. 1 will be described in more detail with reference to FIG. 4.

4 is a conceptual diagram illustrating a proximity depth of a proximity sensor.

As shown in FIG. 4, when a pointer such as a user's finger or pen approaches the touch screen, the proximity sensor 141 disposed in or near the touch screen detects this and outputs a proximity signal.

The proximity sensor 141 may be configured to output different proximity signals according to a distance between the proximity touched pointer and the touch screen (hereinafter, referred to as “proximity depth”).

In FIG. 4, for example, a cross section of a touch screen on which three proximity sensors capable of sensing three proximity depths is disposed is illustrated. Of course, proximity sensors that detect less than three or more than four proximity depths are also possible.

Specifically, when the pointer is completely in contact with the touch screen (d ₀ ), it is recognized as a touch touch. When the pointer is positioned below the distance d ₁ on the touch screen, the pointer is recognized as a proximity touch of a first proximity depth. When the pointer is spaced apart from the distance d ₁ or more and less than d ₂ on the touch screen, the pointer is recognized as a proximity touch of a second proximity depth. When the pointer is spaced apart from the distance d ₂ or more and less than d ₃ on the touch screen, the pointer is recognized as a proximity touch of a third proximity depth. When the pointer is positioned at a distance of more than d ₃ from the touch screen, the proximity touch is recognized as released.

Accordingly, the controller 180 may recognize the proximity touch as various input signals according to the proximity depth and the proximity position of the pointer, and perform various operation control according to the various input signals.

[Description on Figure 5-Transparent Touch]

FIG. 5 is a conceptual diagram illustrating a control method for a touch operation when a pair of display units 156 and 157 overlap.

The terminal disclosed in this drawing is a folder-type terminal in which a folder unit is foldable with respect to a main body. The first display unit 156 mounted on the folder unit may be a light transmissive type or a transparent type such as a TOLED, but the second display unit 157 mounted on the main body may have a form in which light does not penetrate like an LCD. The first and second display units 156 and 157 may be configured as touch screens capable of touch input, respectively.

For example, when a touch (contact touch or proximity touch) with respect to the first display unit or the TOLED 156 is detected, the controller 180 may determine the type of touch and the touch time. Accordingly, at least one image of the image list displayed on the TOLED 156 may be selected or run.

Hereinafter, touch, long touch, long touch & drag (drag) with respect to a method in which information displayed on another display unit or LCD 157 is controlled when a touch is applied to the TOLED 156 exposed to the outside in an overlapped form. The description is based on the input method separated by).

In the overlapped state (mobile terminal is closed), the TOLED 156 is disposed to overlap the lower side of the LCD 157. In this state, when a touch different from a touch for controlling the image displayed on the TOLED 156 is detected, for example, a long touch (for example, a touch lasting more than 2 seconds to 3 seconds), the controller 180 ) Causes at least one image of the image list displayed on the LCD 157 to be selected according to the sensed touch input. The result of the execution of the selected image is displayed in the TOLED 156.

The long touch can also be used to selectively move a desired one of the objects displayed on the LCD 157 (without its execution operation) to the TOLED 156. That is, the user may select one region of the TOLED 156 corresponding to a specific object of the LCD 157. In the long touch, the controller 180 moves the object to the TOLED 156 for display. The object displayed on the TOLED 155 may also be transferred to the LCD 157 according to a predetermined touch input, for example, flicking or swirling, on the TOLED 156. This figure illustrates the case where the number 2 menu displayed on the LCD 156 is moved to the TOLED 156 for display.

If another input, for example, a drag, is additionally detected along with the long touch, the controller 180 is a function related to the image selected by the long touch. Can be displayed on. In this figure, the case where the preview (male photograph) with respect to the 2nd menu (image file) is performed is illustrated.

When the preview screen is output and the drag is made to the TOLED 156 for another image while maintaining the long touch, the controller 180 controls the selection cursor (or selection bar) of the LCD 157. The selected cursor is displayed on the preview screen (woman photo). After that, when the touch (long touch and drag) is finished, the controller 180 displays the first image selected by the long touch.

The touch operation (long touch and drag) is performed when a slide (movement of the proximity touch corresponding to the drag) is detected along with the long proximity touch (the proximity touch lasting at least 2 seconds to 3 seconds) for the TOLED 156. The same applies to.

If a touch operation other than those mentioned above is detected, the controller 180 may operate in the same manner as a general touch control method.

The control method for the touch operation in the overlapped form may be applied to a terminal having a single display. In addition, the present invention can be applied to a terminal different from a folder type having a dual display.

[Description on Figure 6-Haptic]

6A and 6B are views referred to for describing a proximity touch recognition region in which a proximity signal is detected and a haptic region in which a haptic effect occurs.

6A shows an object such as an icon or a menu item in a circle for convenience of description. As shown in (a) of FIG. 6A, an area where an object is displayed on the display unit 151 may be divided into a first area A in the center and a second area B surrounding the area. The first area A and the second area B may be configured to generate a tactile effect having different intensities or patterns. For example, the first vibration may be output when the second region B is touched, and the second vibration may be output when the first region A is touched.

If it is necessary to simultaneously set the proximity touch recognition area and the haptic area in the area where the object is displayed, the haptic area where the haptic effect occurs and the proximity touch recognition area where the proximity signal is detected Can be set differently. That is, the haptic area may be set narrower than the proximity touch recognition area, or the haptic area may be set wider than the proximity touch recognition area. For example, in (a) of FIG. 6A, an area including the first area A and the second area B may be set as a proximity touch recognition area, and the first area A may be set as a haptic area.

As shown in Fig. 6A (b), the area in which the object is displayed is divided into three areas A, B, and C, or as shown in Fig. 6A (c), N (N> 4). It can also be divided into three areas. Each divided area may be configured to generate a tactile effect having different intensities or patterns. Even when the area in which one object is displayed is divided into three or more areas, the haptic area and the proximity touch recognition area may be set differently according to the use environment.

The display unit 151 may be configured such that the size of the proximity touch recognition area varies depending on the depth of proximity. That is, as shown in (a) of FIG. 6B, the corresponding proximity touch recognition area is gradually reduced to 'C', 'B', and 'A' according to the proximity depth to the display unit 151, Alternatively, the corresponding proximity touch recognition region may be gradually increased according to the depth of proximity to the display unit 151. Even in this case, the haptic region may be set to a constant size regardless of the depth of proximity to the display unit 151 as in the 'H' region illustrated in FIG. 6B.

 In the case of dividing the region where the object is displayed for setting the haptic region or the proximity touch recognition region, in addition to the concentric division as shown in FIG. 6A, the division in the horizontal or vertical direction, the radial division, and a combination thereof Various methods, such as division, can be used.

[Description on FIG. 7-Projector Module Form Factor]

A configuration in which the above-described projector module is provided in the bar type mobile terminal 100 as described above will be described with reference to FIGS. 7A and 7B. 7A and 7B are perspective views illustrating an example of a mobile terminal related to the present invention.

To the main body 103 of the mobile terminal, the projector body 105 having the projector module 155 may be coupled to be rotatable.

That is, the projector body 105 may be hinged to the main body 103, and the projection angle is adjusted when the image is projected using the projector module 155 provided in the projector body 105. The camera 121 may be disposed on the projector body 105 to capture an image projected by the projector module 155.

FIG. 7A illustrates a state before the projector body 105 rotatably coupled to the main body 103 is rotated, and FIG. 7B illustrates a state after the projector body 105 is rotated.

According to an embodiment of the present invention, a virtual background image is generated by synthesizing a background image by using a plurality of image frames sequentially input through a camera, and recognizing the position and / or displacement of the central object therewith, A mobile terminal capable of performing a corresponding function and a command input method thereof are provided. This will be described with reference to FIG. 8.

8 is a flowchart illustrating a command recognition method performed in a mobile terminal according to an embodiment of the present invention.

Referring to FIG. 8, first, a user may enter an image recognition input mode using a camera 121 of a terminal through a predetermined menu operation (S210). Here, the image recognition input mode refers to a mode in which a predetermined function corresponding to the input image signal can be performed by recognizing image data input through the camera 121 as a user input signal.

In this case, the image input through the camera 121 may be largely divided into a background image and a central object. Here, the central object is an object that is the core of the image data input through the camera, and may be recognized as another input signal to the controller 180 of the mobile terminal according to its movement, position, or displacement. The central object may be a part of the user's own face or limb when self-photographing through the camera 121 of the mobile terminal, or may be a predetermined stylus pen or the like.

The background image refers to the remaining portion of the image input through the camera 121 except for the center object, and may be recognized as another input signal according to the relative position of the center object.

In addition, the camera 121 of the mobile terminal according to the present invention may receive an image of a predetermined shooting range and provide image data to the controller, and may capture a predetermined number of consecutive images per second. Hereinafter, in the present specification, each image continuously generated by the camera 121 is referred to as an "image frame". The video frame may have a serial number in the order of generation, and the first generated video image is called the first video frame, and then the Nth generated image is referred to as the Nth image frame, and the next generated image is N. This is called the + 1th image frame.

When the image frames including the background image and the center object are sequentially input through the camera 121, two processes may be performed together in the mobile terminal. One of them is a process of generating a virtual background image using the background image of each image frame (S220A), and the other is the position and / or displacement of the central object with respect to the virtual background image generated by the process of S220A in each image frame. The process of determining (S220B). Each of the processes S220A and S220B may be performed by the controller 180 of the mobile terminal.

First, a process of generating a virtual background image using a background image of each image frame (S220A) will be described.

The mobile terminal may first select an arbitrary reference image frame to generate a virtual background image. Here, the virtual background image is a background image first input through the camera 121 as an initial background image, and synthesizes a new background part, which has not been previously, from the background image sequentially input over time, to the initial background image. The background image generated by the method. Therefore, the virtual background image may have a size larger than the predetermined shooting range of the camera 121.

In this case, a method of synthesizing the image pixels of the background image itself may also be used to generate the virtual background image. However, since the data size of the virtual background image may be excessively large, a more efficient generation method is preferably used. To this end, a method of extracting a feature point from a background image of an image frame captured by the camera and synthesizing the feature point may be used.

Here, the feature point refers to a representative pixel or a set of pixels to be compared in each image frame for more efficient comparison. This may mainly be a boundary point having a large fluctuation range such as a line, contrast, or color that is easily detected in an image frame. In order to extract such a feature point from an image frame, an autocorrelation matrix based on information around pixels may be used for some or all of the pixels i of an input image, as shown in Equation 1 below.

는 화소 i를 중심으로 한 사각 윈도우(rectangular window)이며

와

는 각각 x,y 방향으로의 그래디언트(gradient)이다. 자기상관행렬

를 통해서 우리는 행렬의 고유치(Eigenvalue)를 구할 수 있다. 이 고유치가 충분히 크다면 이 화소는 움직임을 추적하기 용이한 화소, 즉 특징점으로 선택될 수 있다. 해당 영상 프레임에서 일부 또는 모든 화소에 대해 자기상관행렬과 고유치를 구하고 고유치가 큰 순서대로 정렬을 한 뒤, 상위 개의 화소를 움직임 추정을 위한 초기 특징점으로 할당할 수 있다.

Is a rectangular window around pixel i

Wow

Are gradients in the x and y directions, respectively. Autocorrelation matrix

We can find the Eigenvalue of the matrix. If this eigenvalue is large enough, this pixel can be selected as a pixel, i. The autocorrelation matrix and the eigenvalues of some or all of the pixels in the corresponding image frame may be obtained, the eigenvalues may be arranged in the order in which the eigenvalues are arranged, and then the upper pixels may be assigned as initial feature points for motion estimation.

However, not all regions of the image frame are suitable as candidates for the feature points. The boundaries of the image lose track points as objects or feature points pass by. For example, suppose that a feature point is detected around a boundary in a previous image. If the pixel moves to the center of the screen, the point does not have any problem in finding the moved pixel in the current image. But if you move off the screen, the tracking will fail. In order to prevent this problem in advance, the feature points are extracted from the region except the outer boundary.

If noisy features are obtained initially, the number of n feature points can be reduced continuously each time the process is repeated. Since securing a certain number of reliable feature points is an important factor of the algorithm, when the first extracted feature points are reduced below a certain ratio or number, the feature points can be reset through an initialization process.

A set of points corresponding to the current image is derived through the feature points extracted through the above-described process. The feature flow method can be used for feature point tracking. Lightflow is an algorithm whose background is tracking the flow of light. To this end, a method of tracking the next feature point by an iterative operation using a predetermined image pyramid may be applied. In this case, although the center object or the background does not move in the image frame input by the actual camera, unintended movement may be detected by an environment in which light changes are severe or by an automatic brightness control function of the camera. In addition, since the light flow is derived from the assumption that the time difference of the input image is very short, an error may occur in the case of very fast movement.

이라 한다면 광흐름 추적 과정을 거쳐 얻어진 각 영상에 대응되는 점의 집합은 아래 수학식 2와 같다.In order to correct the error caused by this, the geometric distance between the point in the previous image and the corresponding point in the next image is calculated. If the value is larger than the average of the other values, it is determined as an error and excluded from the next update. These k points of low reliability are lost and a pair of nk feature points and tracking points are passed to the next step. Each of the two image frames obtained with time difference from the previous process

In this case, the set of points corresponding to each image obtained through the light flow tracking process is represented by Equation 2 below.

를 구할 수 있다.At this time, k feature points among the n feature points are removed due to an error and only the last nk feature points are reflected. These points correspond to each other and the Affine transformation matrix minimizes Equation 3 using the least-squares method.

Can be obtained.

는 Homogeneous 좌표로써

이다. 얻어진 Affine 변환 행렬

는 두 좌표,

로 표현되는 벡터 공간의 관계를 나타내는데 행렬 내의 특정 원소나 그 조합을 통해서 병진(Translation), 축척(Scaling), 회전(Rotation), 층밀리기(Shearing)와 같은 선형 좌표 변환을 감지할 수 있다. 그 중에서 병진 이동을 이용하여 2차원의 가로, 세로 방향의 움직임 벡터는 주어진

에 대해서 다음의 수학식 4 및 5와 같이 각각 얻을 수 있다.here

Is the Homogeneous coordinate

to be. Affine transformation matrix obtained

Is two coordinates,

It represents the relationship of vector space represented by. It can detect linear coordinate transformation such as translation, scaling, rotation, shearing through specific elements or combinations of them in the matrix. Among them, two-dimensional horizontal and vertical motion vectors using translational movement are given

For Equation 4, Equations 4 and 5 can be obtained, respectively.

Motion vectors updated in real time are less stable due to fine noise. Therefore, adjustment of sensitivity is required, and a moving average can be used for that. The moving average is mainly used to analyze a long-term value of a severe change in real time, and stabilizes the value by averaging from the present to a specific point in time as shown in Equation 6 below.

는 시간

에서의 움직임 벡터이며 위에서 구한

나

가 될 수 있다.

은 평균을 구할 시간 범위이다. 여기서

은 매 영상 프레임을 통하여 업데이트 되는 결과가 불안정하지 않을 만큼 충분히 크고, 실시간으로 변화하는 값을 즉각적으로 보여주기에 충분할 만큼 작은 것이 바람직하다. 위와 같은 방법으로 구해진 이동 평균은 기 설정된 갯수의 방향, 예를 들면 상(up), 하(down), 좌(left), 우(right) 및 멈춤(Idle)의 다섯 가지 방향 중 하나로 대응되어 인식될 수 있다.

Time

The motion vector at, obtained from

I

Can be

Is the time range to average. here

It is desirable that the result of updating through each image frame is large enough not to be unstable and small enough to immediately show the value changing in real time. The moving average obtained by the above method is recognized in correspondence with one of five preset directions: for example, up, down, left, right, and idle. Can be.

An example in which the above-described feature point extraction and the determination of the displacement through the moving average are actually applied will be described with reference to FIG. 9. 9 illustrates a feature point and a displacement tracking simulation result thereof according to an embodiment of the present invention.

Referring first to FIG. 9A, feature points extracted through the above-described method from the still image of the cup are indicated by red dots on the left side. On the right side, when the cup is moved upward, the feature point is traced through the moving trace of the red dot, and the moving direction and the degree are displayed.

In addition, (b) of FIG. 9 shows a case in which the same method as in (a) of FIG. 9 is used, but the image of the bulletin board is moved to the left.

So far, the method of extracting feature points and comparing the image frames with reference to Equations 1 to 6 has been described. Referring to FIG. 8 again, the process of S220A will be described.

The mobile terminal can extract the feature point of the background image through the above-described method using the N-th image frame as the reference image frame (S221). In this case, the N-th image frame, which becomes the reference image frame, may be the first image frame input through the camera 121 or may be arbitrarily set according to a user's menu operation or preset settings.

When the feature point of the background image is extracted from the N-th image frame, the controller 180 of the mobile terminal can generate an initial virtual background image that is the basis of the virtual background image using the same (S222). The feature point and the initial virtual background image of the background image extracted from the N-th image frame may be stored in the memory unit 160.

Next, when an N + 1 th image frame is input through the camera 121, the controller may extract a feature point of the background image therefrom (S223).

When the feature point is extracted from the background image of the N + 1th image frame, the controller 180 of the terminal may compare the feature point with the feature point extracted from the background image of the Nth image frame (S224). This comparison process may be performed through the above-described methods such as light flow tracking and moving average. In this case, the feature point of the background image of the N-th image frame may use information stored in the memory unit 160, and the feature point extracted from the background image of the N + 1th image frame may also be stored in the memory unit 160.

As a result of performing the comparison process, the controller 180 may determine whether there is a new background image that does not exist in the Nth image frame in the N + 1th image frame (S225).

As a result of the determination, when a new background image exists, the controller 180 may generate a larger virtual background image by synthesizing the portion with the initial virtual background image generated in step S222.

Next, a process of determining the position and / or displacement of the central object with respect to the virtual background image (S220B) will be described.

The mobile terminal can extract the feature point of the central object through the above-described method using the N-th image frame as the reference image frame (S221B). In this case, the N-th image frame, which becomes the reference image frame, may be the first image frame input through the camera 121 as in the above-described process of generating the virtual background image, and may be a menu operation or a preset setting of the user. It may be set arbitrarily according to the. The extracted feature points of the central object may be stored in the memory unit 160.

Next, when the N + 1 th image frame is input through the camera 121, the controller 180 can extract the feature point of the central object therefrom (S222B).

When the feature point of the center object is extracted from each image frame through the above process, the controller 180 determines the position of the feature point of the center object extracted from the N + 1th image frame. It can be compared with the position (S223B). This comparison process may be performed through the above-described methods such as light flow tracking and moving average. In this case, the position of the feature point of the center object extracted from each image frame and the result of the comparison process S223B may be stored in the memory unit 160.

Through the comparison process S223B, the controller may calculate a vector value of the movement of the central object, and determine the current position and / or displacement of the central object with respect to the virtual background image generated through the S220A process (S220A). S224B). That is, the controller 180 does not calculate the displacement based on the current position of the center object or the position of the center object in the N-th image frame based only on the background image included in the N + 1th image frame. Instead, the controller calculates the current position and / or displacement of the central object based on the entire virtual background image.

Therefore, assuming that at least some of the feature points of the central object do not exist in any image frame, it is difficult to determine the position of the central object on the background image included in the frame. However, if the position of the feature point of the central object in the previous image frames stored in the memory unit 160 or the direction and degree of movement are used, the position of the central object is estimated on the virtual background image even if the position does not exist on the current image frame. Or predictable. As a result, this method can more stably track the central object in the image recognition input mode and can be recognized as a user input.

Subsequently, the controller 180 may perform a predetermined function corresponding to the virtual background image generated in the above-described steps S220A and S220B and the position and / or displacement of the central object.

For example, it is assumed that a predetermined cursor is present on the display unit 151, and a predetermined application is executed such that the cursor moves in accordance with the movement of the central object in the image frame input through the camera 121. At this time, if the position of the center object in the N + 1th image frame is moved to the right side than the position in the Nth image frame, the controller 180 recognizes this as a user input and displays a predetermined cursor ( Cursor) can be controlled to move to the right.

As another example, when the central object stays at a specific position for a predetermined time, a predetermined function corresponding thereto may be performed.

The predetermined function may be performed according to the position and / or displacement of the central object in one image frame interval, but may be performed after monitoring the position and / or displacement of the central object in various image frame intervals. It may be.

Next, as in the process of S220A and S220B, the N + th image frame is compared with the N + 1st image frame, and the same process is performed by using the N + 1st image frame as the comparison target of the N + 2th image frame. In order to add 1 to N (S240). In this case, not only 1 but also other natural numbers may be added to the frame number N according to the system setting or the need of the mobile terminal.

On the other hand, the controller 180 of the mobile terminal according to the present invention can more stably respond to the movement of the central object as the virtual background image is larger, but if larger than a certain range according to the available resources of the mobile terminal of the memory unit 160 It may act as a burden on the calculation of the capacity or the controller 180. Therefore, the controller 180 satisfies a condition for stopping generation of a preset virtual background image, such as the generated virtual background image satisfies a predetermined size or starts generating the virtual background image for a predetermined time or exceeds the number of frames N. It may be determined whether or not (S250).

When the controller 180 determines that the preset generation stop condition is satisfied, the controller 180 does not perform the step S220A anymore and re-executes only the step S220B to continue tracking only the central object.

On the contrary, the controller performs steps S220A and S220B together to track the central object and add a new portion of the background image input through the camera 121 to the existing virtual background image to accumulate a larger virtual background. You can create an image.

Meanwhile, in order to distinguish the background image from the central object in the image frame input through the camera 121, the controller divides the feature points extracted from each image frame into two groups based on similar displacements. This can be used. In more detail, when all the feature points move in the same direction, the controller may recognize that the photographing direction of the camera 121 is changed. The feature point of one group does not have a large displacement, but if there is a displacement of another group, the displaced group may be recognized as a central object. In this way, once the displacement of the central object is extracted, the rest of the center object may be recognized as a background image.

However, since the above-described method may be inaccurate depending on the situation, a method for more accurately distinguishing the background image from the center object in the image frame will be described with reference to FIG. 10. FIG. 10 is a display state diagram in which a predetermined image for receiving a center object is displayed in a mobile terminal according to an embodiment of the present invention. FIG.

Referring to FIG. 10A, when the central object is assumed to be at least a part of the upper body of a person, a predetermined object for recognizing the central object is displayed on the display unit 151 of the mobile terminal as the image recognition input mode is entered. An image may be displayed. The predetermined image for recognizing the central object may include a portion 310 covering the background image and a portion 320 in which the central object is to be located. In this case, the image that is input through the camera 121 may be blocked at the portion covering the background image, and the image input through the camera 121 may be displayed only at the portion 320 at which the central object is to be located. In this case, it is assumed that the central object is the upper body of the person, and the portion 320 in which the central object is to be located is preferably prepared in the form of the upper body of the person. In this case, when the upper body 330 of the person who becomes the center object enters the inside of the part 320 in which the center object is to be located, as shown in FIG. Using only the image of) can extract the feature point of the central object more accurately. The predetermined image for recognizing the central object may be prepared in various forms according to the type of the object to be recognized by the user as the central object.

Next, referring to FIG. 11, a method of estimating a location using a virtual background image when the central object deviates from the image frame will be described. 11 illustrates an image actually input through a camera and a feature point extracted using the same in a terminal according to an embodiment of the present invention.

First, referring to FIG. 11A, a feature point is extracted from an image frame received through the camera 121, and a feature point corresponding to a background image is a point, and a feature point corresponding to the center object 400 is diamond. Each is marked with a type. The right figure shows an example of an initial virtual background image that is actually recognized by the controller of the mobile terminal, and feature points are displayed.

Next, referring to FIG. 11B, the center object 400 moves to the left side along with the photographing direction of the camera 121 in the situation of FIG. ), It can be seen that a new feature point 420 has been extracted. Accordingly, a new background image portion 410 is added to the virtual background image on the right side, so that the new feature point 420 is displayed. In this case, the virtual background image may not appear in the current image frame, but the portion 430 existing in the initial virtual background image is included.

In this state in which a background image larger than the initial virtual background image shown in FIG. 11A is generated, as shown in FIG. Consider the case where only is detected.

In this case, the conventional method fails to track the central object. However, according to the present invention, although the center object is not detected in the image frame itself 450 as shown in FIG. 11 (d), the position on the virtual background image 460 is based on the movement pattern of the center object extracted from previous image frames. Because we can estimate, we can reliably track the center object. In other words, even if only a part of the center object is detected or disappeared from the image frame, if the center object is detected from the image frame again before a certain frame elapses, the mobile terminal does not need to recognize the center object and the background image from the beginning and do not need to recognize it. It can work as

However, when the central object is not continuously detected in the image frame for a predetermined frame or more, the generated virtual background image may be initialized and generation of the virtual background image may be resumed when the center object is detected again.

Next, a method of completing generation of a virtual background image having a predetermined range in advance will be described with reference to FIG. 12. 12 is a conceptual diagram illustrating a method for receiving a virtual background image in advance in a terminal according to an embodiment of the present invention.

In FIGS. 12A to 12C, the mobile terminal recognizes a feature point of the central object and generates a 3X3 sized virtual background image by receiving a background image in eight directions around the first received background image. Assume the case. Also, the left figure shows an image displayed on the display unit 121 of the mobile terminal, and the right figure shows a virtual background image 510 generated by the mobile terminal. By the way, the mobile terminal according to an embodiment of the present invention will recognize the virtual background image as an array of feature points extracted from each image frame. However, in this figure, the virtual background image is displayed in the form of a simple figure.

First, as shown in the left figure of FIG. 12A, an image frame including a central object 500 and a background image input through the camera 121 may be displayed on the display unit 151 of the mobile terminal. The mobile terminal may generate an initial virtual background image 511, which is a base of the virtual background image 510 of a preset size, according to the input image frame. The display unit 151 may display a predetermined direction indication 501 for the mobile terminal to sequentially receive a background image of a preset range.

Next, as shown in FIG. 12B, when the user changes the photographing direction of the camera 121 according to the direction indication 501, the mobile terminal may add a new background image 512 to the virtual background image 510. . In addition, the display unit 151 may display a direction indication 502 to receive the next background image.

In the same manner, in FIG. 12C, the background image 513 of the next direction may be added to the virtual background image 510, and the direction indication 503 may be displayed on the display unit 151.

If this process is repeated for each of the remaining directions, the mobile terminal can generate a virtual background image having a desired size. In this case, the mobile terminal may stop generating the virtual background image and perform only a process of determining the position and / or displacement of the central object by using the previously generated virtual background image.

Here, the size, shape, generation order, and direction indication form of the virtual background image to be generated in advance are exemplary and may be changed as necessary.

In the image recognition input mode described so far, a function that can be executed through a recognized user input according to the position and / or displacement of the central object with respect to the virtual background image includes moving, selecting, and manipulating a cursor or menu on a predetermined application. It may be possible. It can also be used for triggering the shutter when taking self-portraits using the camera, to recognize sign language or to manipulate games.

These functions are exemplary, and the mobile terminal having the camera described above is not limited to the configuration and method of the above-described embodiments, but the embodiments may be modified in various ways. All or some of these may optionally be combined.

Further, according to an embodiment of the present invention, the above-described method can be implemented as a code that can be read by a processor on a medium on which the program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may be implemented in the form of a carrier wave (for example, transmission over the Internet). Include.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

Figure 2a is a front perspective view of a mobile terminal according to an embodiment of the present invention.

2b is a rear perspective view of a mobile terminal according to one embodiment of the present invention;

3A and 3B are front views of a mobile terminal for explaining an operation state of the mobile terminal according to the present invention.

4 is a conceptual diagram for explaining a proximity depth of a proximity sensor.

5 is a conceptual diagram illustrating a control method for a touch operation in a form in which a pair of display units overlap.

6A and 6B are conceptual views for explaining a proximity touch recognition region where a proximity signal is detected and a haptic region that generates a tactile effect, respectively.

7A and 7B are perspective views of a mobile terminal according to an embodiment of the present invention.

8 is a flowchart illustrating a command recognition method performed in a mobile terminal according to an embodiment of the present invention.

9 is a simulation result of the feature point and its displacement according to an embodiment of the present invention.

FIG. 10 is a display state diagram in which a predetermined image for receiving a center object is displayed in a mobile terminal according to an embodiment of the present invention; FIG.

11 is an image actually input through a camera in the terminal according to an embodiment of the present invention and the feature points extracted using the same.

12 is a conceptual diagram illustrating a method for receiving a virtual background image in advance in a terminal according to an embodiment of the present invention.

Claims (20)

A camera for continuously photographing a plurality of image frames including a background image and a center object within a predetermined shooting range; And In the image recognition input mode, using the plurality of image frames, each image frame is sequentially compared with a previous frame starting from a predetermined reference image frame, and a new background image which is not present in the background image of the previous frame is transferred. And a controller configured to generate a virtual background image by accumulatively synthesizing so as to be connected with the background image. The method of claim 1, The control unit, And extracting at least one of a position and a displacement of the center object with respect to the generated virtual background image and recognizing it as a user input, and controlling to perform a function corresponding to the recognized user input. The method of claim 1, The control unit, And comparing feature points of a background image and a center object from each frame, and controlling the extracted feature points to be compared with each other. The method of claim 3, wherein The control unit, And when the center object is not detected in the image frame photographed by the camera, controlling the position of the center object to be estimated on the virtual background image using a movement pattern of the center object. The method of claim 3, wherein Further comprising a memory unit, The controller controls the virtual background image to be stored in the memory unit. The method of claim 5, The control unit, And when the central object or the feature point cannot be continuously detected from the predetermined number of image frames, controlling to delete a virtual background image previously stored in the memory unit and to generate a new virtual background image. The method of claim 1, The predetermined reference frame, And a frame photographed by the camera for the first time in the user input mode. The method of claim 1, The control unit, When the user enters the user input mode, a background image of a preset range is first inputted through the camera to configure an initial virtual background image, and control to sense displacement of the central object using the configured virtual background image. Mobile terminal characterized in that. The method of claim 8, Further comprising a display, The control unit, And a predetermined direction indication to sequentially receive the background image of the preset range is displayed on the display unit. The method of claim 8, The control unit, The mobile terminal, characterized in that for receiving the background image of the surrounding eight directions based on the background image of the first image frame, the initial virtual background image is generated. The method of claim 1, Further comprising a display, The control unit, When entering the user input mode, the display unit displays a predetermined image for recognizing the central object, and displays an image frame of the central object input through the camera and a predetermined image for recognizing the central object. The mobile terminal, characterized in that for controlling the recognition so that the central object first. Generating a virtual background image by extracting a background image of the N-th image frame; Extracting a background image of an N + α-th image frame and comparing it with a background image of the N-th image frame; And And adding a new portion not included in the N-th image frame to the virtual background image in the background image of the N + α-th image frame as a result of the comparison. The method of claim 12, Extracting a central object from each of the image frames; And And determining a position and a displacement of the extracted central object with respect to the virtual background image. The method of claim 13, And performing a predetermined function corresponding to the determined position or displacement of the central object. The method of claim 13, Predicting the position of the center object with respect to the virtual background image by using the displacement detected in the previous frame of the frame in which the center object is not detected if the center object is not detected in any one of the respective image frames; Further comprising a command input method of the mobile terminal. The method according to any one of claims 12 to 15, And the N-th image frame and the N + α-th image frame are sequentially photographed through a camera. The method according to any one of claims 12 to 15, The comparing step, The command input method of the mobile terminal, characterized in that performed by a method of comparing the extracted feature points of each image frame with each other. The method according to any one of claims 12 to 15, The comparing step and the adding step are performed again until the predetermined condition is satisfied while adding [alpha] to N, and the command input method of the mobile terminal is characterized in that it continues. The method of claim 18, The preset condition is And a case in which a predetermined time elapses, a preset image frame number is reached, and the virtual background image reaches a predetermined size. The method of claim 18, Α is Command input method of a mobile terminal, characterized in that the natural number.

Images