KR100466855B1 - Method and System for Providing User Interface by Using Image Signal for Mobile Communication Terminal Equipped with Camera Function - Google Patents

Abstract

PURPOSE: A user interface method and a system using a pickup signal of a camera-attached mobile communication terminal having a photographing function are provided to conduct a pointer operation control similar to a mouse pointer on a display module by using a pickup input of a camera-attached terminal, thereby supplying a user interface similar to a GUI environment of a PC to mobile communication terminal users. CONSTITUTION: An image signal is generated by photographing an object with a pickup unit(S300). An encoder module converts the photographed image signal into image data by using the pickup unit(S302). A movement of the object is detected through the image data inputted to a detection module or raw data directly inputted to the detection module by being bypassed in the encoder module(S304). An operation detected under control of a CPU is finally displayed on a display module while a specific function is executed(S306).

Description

Method and System for Providing User Interface by Using Image Signal for Mobile Communication Terminal Equipped with Camera Function}

The present invention relates to a user interface method and system using an imaging signal of a mobile communication terminal with a camera having a photographing function. More specifically, an image signal photographed using a mobile communication terminal with a camera having a photographing function is converted into image data using an encoder module, and a detection module receives the image signal or the image data as an input and moves a subject. The CPU relates to a user interface method and system for controlling to implement a detected operation on a display module of a mobile communication terminal.

As computers, electronics, and communication technologies have advanced dramatically, various wireless communication services using wireless networks have been provided, and functions of mobile communication terminals have been gradually expanded. The most basic wireless communication service is a wireless voice call service that provides a voice call to a mobile terminal user wirelessly, which can provide a call service regardless of time and place. In addition, while providing a text message service to compensate for the shortcomings of the voice call service, a wireless Internet service has emerged to provide an Internet communication service to the user of the mobile communication terminal through a wireless communication network. In recent years, models equipped with imaging means such as digital still cameras and digital video cameras have become popular in portable information terminals such as laptop personal computers, PDAs, and cellular phones.

Accordingly, the subscribers of the mobile communication service can freely move and talk with the other party anytime and anywhere using the wireless communication service, and also provide various information necessary for life through the wireless Internet service, such as news, weather, sports, securities, Exchange rates or traffic information may be provided in various forms such as text, voice, or image. In addition, various applications such as games and word processors can be used on a mobile communication terminal. Also, a mobile communication terminal equipped with an image capturing means can capture still images or moving images of a subject, and transfer and exchange image information and make a video call. .

Accordingly, customers of mobile communication services require mobile terminals capable of providing various additional services in addition to the basic voice call service, and mobile communication service providers and manufacturers of mobile communication terminals are required to add new additional services to meet the needs of these customers. While developing services, a mobile communication terminal capable of supporting these services is being developed.

1 is a perspective view showing the appearance of a conventional mobile communication terminal with a camera function.

As shown in FIG. 1, a mobile communication terminal 100 having a camera function generally includes a small camera 110 at a location convenient for photographing a subject. 1 shows a so-called camera built-in mobile communication terminal 100, but the small camera 110 is not detachably mounted to the mobile communication terminal 100 in a detachable form so that it can be used by connecting to the mobile communication terminal 100. It may be configured.

After the user selects a camera shooting function among a plurality of menus displayed on the mobile communication terminal, the user fixes the small camera 110 provided in the mobile communication terminal 100 toward a subject to be photographed and executes a shooting operation. The mobile communication terminal 100 stores the image photographed by the small camera 110 as image data in a digital format. Subsequently, the user may transmit the image data stored in the mobile communication terminal 110 to the counterpart wirelessly using a mobile communication network, or move the image data to a computer and edit the image data. However, the conventional mobile communication terminal 100 does not have a function of directly using an imaging signal for a user interface.

When a user uses various applications such as a game and a word processor on a mobile communication terminal, a user generally performs a function selection and operation control using the key input unit 120 of the mobile communication terminal. The user interface is slower in response to the interface and has limited menus that can be used on the key input unit 120, compared to a pointer motion control technology using a mouse in a GUI (Graphic User Interface) environment of a personal computer (PC). There are disadvantages. Accordingly, there are limitations in developing or using various applications for mobile communication terminals.

In order to implement a convenient user interface using the pointer motion control technology, most mobile communication terminals such as PDAs (Personal Digital Assistants) use touch screens. In this case, however, first, the size of the touch screen must be greater than a certain limit in order to control the pointer operation, and secondly, the price of the mobile communication terminal is expensive due to the addition of a display device such as a touch screen. In order to provide additional equipment such as a mouse for a communication terminal, there has been a problem of sacrificing portability.

In order to solve this problem, the present invention converts an image signal photographed using a mobile communication terminal with a camera having a photographing function into compressed image data using an encoder module, and the detection module converts the image signal or the image data. It is an object of the present invention to provide a user interface method and system for detecting movement of a subject as an input and controlling the detected motion to be implemented on a display module of a mobile communication terminal.

1 is a perspective view showing the appearance of a mobile communication terminal having a conventional camera function;

2 is a block diagram schematically illustrating a user interface system according to a preferred embodiment of the present invention;

3 is a flowchart illustrating a user interface method according to an embodiment of the present invention;

4 is a diagram illustrating an example of a movement of a pointer using a user interface method according to an embodiment of the present invention;

5 is a diagram illustrating an example of selecting a pointer using a user interface method according to an exemplary embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

200, 520: imaging means 210: encoder module

220: detection module 230: central processing unit (CPU)

240, 410, 530: display module 510: mobile communication terminal

400, 500: Subject

According to a first object of the present invention, in a user interface method using an imaging signal of a mobile communication terminal with a camera having a photographing function including an imaging means, an encoder module, a detection module, a display module, and a central processing unit, (a) Converting the image signal photographed using the image pickup means into image data in the encoder module; (b) detecting the movement of a subject by receiving the image signal or the image data as an input from the detection module; And (c) implementing the motion detected in step (b) on the display module under the control of the CPU. Provides an interface method.

According to a second object of the present invention, there is provided an application driving method in a mobile communication terminal with a camera having a photographing function including an imaging means, an encoder module, a detection module, a display module, and a central processing unit, comprising: (a) the imaging means; Converting the image signal photographed by using the encoder module to the image data; (b) detecting the movement of a subject by receiving the image signal or the image data as an input from the detection module; (c) implementing the operation detected in the step (b) on the display module under the control of the CPU; And (d) executing the application using the user interface method implemented in step (c). The imaging method includes an image pickup means, an encoder module, a detection module, a display module, and a central processing unit. An application driving method in a camera-equipped mobile communication terminal having a function is provided.

According to a third object of the present invention, a user interface system using an imaging signal of a mobile communication terminal with a camera having a photographing function, comprising: imaging means for photographing a subject to generate an image signal; An encoder module for converting the captured image signal into image data; A detection module configured to detect an operation of a subject by receiving the image signal or the image data as an input; A display module displaying an operation of the subject detected by the detection module; And a central processing unit configured to control an operation of the subject detected by the detection module to be implemented on the display module. do.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are used as much as possible even if displayed on different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

2 is a block diagram schematically illustrating a user interface system according to a preferred embodiment of the present invention.

As shown in FIG. 2, a user interface system using an imaging signal of a mobile communication terminal with a camera having a photographing function according to an exemplary embodiment of the present invention includes an imaging means 200, an encoder module 210, and a detection method. The module 220 may include a central processing unit (CPU) 230, a display module 240, and the like.

Mobile communication terminal with a camera having a shooting function using a user interface system according to the present invention is a PDA (Personal Digital Assistant), cellular (Cellular) phone, PCS (Personal Communication Service) phone, handheld PC (Hand-PC) Held PC), Global System for Mobile (GSM) phone, Wideband CDMA (W-CDMA) phone, EV-DO phone, EV-DV (Data and Voice) phone, Mobile Broadband System (MBS) phone, and the like. . Here, MBS phone refers to a mobile communication terminal to be used in the fourth generation system that is currently discussed.

The imaging means 200 according to the preferred embodiment of the present invention photographs a subject to generate an image signal. Here, the imaging means 200 may include a digital still camera or a digital video camera mounted on the mobile communication terminal. The imaging unit 200 converts an optical image of the subject into an electrical image signal using a photosensitive sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and an analog image using an A / D converter. The video signal may be generated by converting the signal into a digital video signal. That is, the above-described image signal refers to a digital image signal, and the image signal has image information for recognizing the shape of the subject in the user interface system of the present invention.

The encoder module 210 according to the preferred embodiment of the present invention converts the image signal photographed by the image pickup means 200 into compressed image data. Here, the encoder module 210 may be independent hardware that performs an image compression or motion picture compression function, software implemented on the independent hardware, or software mounted on the central processing unit 230. It may include. Image compression or video compression is an operation of reducing the amount of image data without losing the original information for the purpose of storage and transmission, thereby reducing the throughput of the image data.

However, it is also possible to detect the movement of the subject by receiving the captured image signal as an input from the detection module 220 without performing compression conversion on the encoder module 210. The reason for converting the image signal into the compressed image data is to reduce the throughput of the image data, and since the image signal itself has all the image information for recognizing the shape of the subject, the image signal is directly input from the detection module 220. Even if it detects the movement of a subject, it does not deviate from the essential characteristic of this invention. In this case, since the throughput of the image data is increased as compared with the case of using the encoder module 210, it is preferable to have a central processing unit 230 capable of high-speed arithmetic processing and a large image memory for storing the image data.

On the other hand, the encoder module 210 according to the support of the user interface system of the present invention, Graphic Interchange Format (GIF), Joint Photographic Expert Group (JPEG), Bit Map Format (BMP) and Motion Picture Experts Group (MPEG) Can be converted into a compressed video data format.

Here, the GIF format is a video file format developed by CompuServe of the United States in 1987 for the purpose of compressing the image on the Internet and transmitting it quickly. The image that can be saved is limited to 256 colors. The BMP format is a video data format used as a standard in Windows, and is a bitmap format that expresses an image based on pixels. JPEG format is a joint standardization of ISO / IEC JTC 1 (ISO / IEC Joint Technical Committee 1 on Information Technology) and ITU-T (International Telecommunication Union Telecommunication Standardization Sector), which compresses, records or transmits image information. This is how you play it. MPEG is standardized by ISO / IEC JTC 1. It is a method of compressing and encoding digital color video and audio signals and multiplexing encoded signals.

The detection module 220 according to the preferred embodiment of the present invention receives the raw data directly from the image pickup means 200 without passing the image data or the encoder module 210 converted by the encoder module 210 and receives the movement of the subject. Will be detected. In this case, the detection module 220 may include independent hardware for performing a motion detection function of a subject, software implemented on the independent hardware, or software mounted on the central processing unit 230. The software or hardware included in the detection module 220 receives the image data in a continuous shooting order and displays a difference between pixel values at corresponding positions of the image data captured in the previous sequence and the image data captured in the subsequent sequence. You can decide. Through this, the data portion representing the subject of the image data can be distinguished and the movement trajectory of the subject can be tracked so that the subject moves in a different direction while maintaining a constant distance from the imaging means 200 or the distance from the imaging means 200 The motion of the subject may be detected, such as moving in a direction that is approaching.

On the other hand, since the interface system of the present invention can be used to drive various applications in the mobile communication terminal, which will be described later, the detection module 220 can be used to detect a user's hand movement, particularly during movement of a subject.

The CPU 230 according to an exemplary embodiment of the present invention displays the user's hand motion detected by the detection module 220 on the display module 240 of the mobile communication terminal. That is, the CPU 230 maps the detected hand motion to the screen space on the display module 240 using the pointer. Here, the pointer refers to a symbol on the screen, which is controlled by an input tool such as a mouse and used to designate a point or select a point that can be positioned on the screen, and may have an arrow shape or various other shapes. When a mode for using the user interface system of the present invention is selected, the pointer is positioned at the origin of the screen space on the display module 240. Here, the origin may be located at the center of the screen space or may be located at other points according to the user's selection.

When the detection module 220 detects the motion of the subject by analyzing the image data by software or hardware, the detection module 220 moves the previous pointer position to map the detected motion to the screen space on the display module 240. The position coordinates to which the pointer should move may be calculated and provided to the CPU 230 as an input, or the position coordinates to which the pointer should move may be calculated from an operation in which the CPU 230 is directly detected. The CPU 230 may control the display module 240 to update the position of the pointer with the position coordinates to which the pointer should move, and move the pointer to the updated pointer position on the display module 240.

The display module 240 according to the exemplary embodiment of the present invention displays the motion detected by the detection module 220 in the screen space on the display module 240 under the control of the CPU 230. The display module 240 may include a super twisted nematic liquid crystal display (STN LCD), a thin film transistor liquid crystal display (TFT LCD), a plasma display panel (PDP), an organic electroluminescence display (ELD), and the like.

Here, STN LCD is a liquid crystal display device in which an array direction of thin and long liquid crystal molecules is arranged at an angle of 180 degrees between two glass substrates. The TFT LCD consists of a bottom plate on which thin film transistors and pixel electrodes are arranged, a top plate composed of color filters and common electrodes for displaying colors, and a liquid crystal filled between the two glass substrates. It is a liquid crystal display device in which polarizing plates which linearly polarize visible light (natural light) are attached. The PDP encloses a gas such as neon in a narrow gap between two thin glass substrates, and arranges transparent electrodes on the inner surface of the glass in a horizontal direction and a vertical direction to charge the pixel at a point where a pair of charged electrodes meet. (Pixel) is a display device that emits light by gas ionization. The organic ELD is a display device capable of emitting light at a point where these electrodes meet by injecting a fluorescent material between electrodes arranged in a horizontal direction and a vertical direction, and flowing a current through the horizontal electrode and the vertical electrode. .

3 is a flowchart illustrating a user interface method according to a preferred embodiment of the present invention.

First, a mobile communication terminal with a camera having a photographing function should be set to a mode for using a user interface method according to a preferred embodiment of the present invention. That is, when the user is using other services such as voice service in the mobile communication terminal, the mobile communication terminal is set to a mode that does not use the user interface system of the present invention, and in order to use the user interface method of the present invention, The communication terminal must be set to the usage mode. The mode setting method is a method of using any one of the key input unit of the mobile terminal for mode setting, a method of moving to a mode setting menu by using a direction key button from the menu on the mobile terminal, or using the interface system of the present invention. And a method of automatically switching to a use mode when driving various applications.

When switching to the mode for using the user interface method of the present invention, by using the imaging means 200 to photograph the subject to generate an image signal (S300). Next, the image signal photographed using the image pickup means 200 is converted by the encoder module 210 into image data (S302). The movement of the subject is detected through raw data input to the detection module 220 and bypassed by the encoder module 210 and directly input to the detection module 220 (S304). The movement of the subject is generally a user's hand movement. Finally, the operation detected by the central processing unit 230 is displayed on the display module 240 to execute a specific function (S306).

The specific function may include a moving function of moving a pointer on the display module 240 by a motion of a detected subject, or a selection function of executing a function of transmitting, storing, or deleting specified in an icon indicated by the pointer. It may include.

4 is a diagram illustrating an example of a movement of a pointer using a user interface method according to an exemplary embodiment of the present invention.

As shown in FIG. 4, when the subject 400 moves in a circle in a clockwise direction, the imaging unit 200 continuously photographs the movement of the subject 400. When the encoder module 210 converts the image signal into image data in a continuous shooting order, and detects the movement of the subject 400 by analyzing the image data in the detection module 220, the pointer in the central processing unit 230 is The position of the pointer is updated with the position coordinates to be moved, and the display module 410 is controlled to move the pointer to the position of the updated pointer on the display module 410. Therefore, the pointer on the display module 410 also moves in a circle in a clockwise direction.

5 is a diagram illustrating an example of selecting a pointer using a user interface method according to an exemplary embodiment of the present invention.

As shown in FIG. 5, when the subject 500 moves vertically with respect to the imaging means 520, that is, when the subject 500 moves from top to bottom with respect to the imaging means 520 as if the mouse button is pressed, the subject 500 is taken by the imaging means. When approaching 520, the size of the subject 500 recognized by the imaging unit 520 increases, and the detection module 220 may detect image data by analyzing the image data. The central processing unit 230 controls the display module 530 to click on the icon that the pointer currently points on the display module 530, and executes a selection operation such as transmission, storage, and deletion set in the icon. Done.

In addition, for the convenience of users of the user interface method of the present invention, by pressing any one of the designated button of the key input unit of the mobile terminal is recognized as the operation of selecting the icon that the pointer is currently pointing to execute the above-described selection operation It is also possible to set it up.

Meanwhile, various applications may be driven on a mobile communication terminal using a user interface method and system using an imaging signal of a mobile communication terminal with a camera having a photographing function according to an exemplary embodiment of the present invention. That is, by using the user interface method and system of the present invention, it is possible to provide a pointer operation control similar to the GUI environment of a PC in a mobile communication terminal environment, so that an application requiring complicated pointer manipulation can also be driven and operated on a mobile communication terminal. Can be.

Here, an application is a computer program designed to directly perform a specific function, also called an application program. For example, in the case of inventory management with a computer, a specific function is inventory management and the application executing the application is an application. In this respect, an application is distinguished from an operating system that runs a computer or a utility that assists in executing a specific function and supports the maintenance of a computer system. The application can manipulate text, numbers, or figures, or a mixture of all of these elements, depending on the task being processed. Document processors like MS-Word and Hangul, spreadsheets like Excel, database management systems like Oracle, computer games, mobile games, and browsers can all be included in applications.

The above description is merely illustrative of the present invention, and those skilled in the art to which the present invention pertains may various modifications without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed herein are not intended to limit the present invention but to describe the present invention, and the spirit and scope of the present invention are not limited by these embodiments. It is intended that the scope of the invention be interpreted by the following claims, and that all descriptions within the scope equivalent thereto shall be construed as being included in the scope of the present invention.

As described above, according to the present invention, it is possible to control a pointer operation similar to a mouse pointer on the display module by using an imaging input of a mobile terminal with a camera having a photographing function, so that a GUI (Graphic User) of a personal computer (PC) can be controlled. Interface) By providing a user interface similar to the environment to users using a mobile communication service, there is an effect that can conveniently operate and manipulate various applications.

In addition, there is no need to install additional hardware equipment such as a touch screen in the mobile communication terminal to provide the user interface as described above, there is an effect that can increase the portability.

Claims (17)

A user interface method using an imaging signal of a mobile communication terminal with a camera having a photographing function including an imaging means, an encoder module, a detection module, a display module, and a central processing unit, (a) converting the image signal photographed using the imaging means into image data in the encoder module; (b) detecting an operation of a subject by receiving the image signal or the image data as an input from the detection module; And (c) implementing the operation detected in the step (b) on the display module under the control of the CPU; A user interface method using an image pickup signal of a mobile communication terminal with a camera having a shooting function comprising a. The method of claim 1, The user interface method using an image pickup signal of a mobile terminal with a camera having a shooting function, characterized in that detecting the operation of the user's hand or finger in the step (b). The method of claim 2, In the step (c), by detecting the motion of the hand or the finger, converting the detected motion into the motion of the pointer on the display module to implement the movement or selection of the pointer. A user interface method using an image pickup signal of a communication terminal. The method of claim 1, The detection module calculates a position coordinate to which the pointer should move from a previous pointer position and provides it to the CPU as an input, or the CPU directly calculates a position coordinate to which the pointer should be moved. A user interface method using an image pickup signal of a mobile communication terminal with a camera having a shooting function, characterized in that the position is updated. The method according to claim 3 or 4, If it is detected by the detection module that the subject approaches the image pickup means in the vertical direction, it is determined that the selection is made, and it is recognized as a selection operation, and is set on an icon that the pointer points on the display module. A user interface method using an image pickup signal of a mobile communication terminal with a camera having a shooting function, characterized by executing a selection operation. The method according to claim 3 or 4, Pressing any one button designated by the key input unit of the mobile communication terminal recognizes the selection and executes the selection operation set on the icon indicated by the pointer on the display module. A user interface method using an imaging signal of a mobile communication terminal with a camera having a shooting function. The method of claim 1, The image data format may be at least one of a Graphic Interchange Format (GIF), a Joint Photographic Expert Group (JPEG), a Bit Map Format (BMP), and a Motion Picture Experts Group (MPEG). User interface method using the image pickup signal. A method of driving an application in a camera-equipped mobile communication terminal having a photographing function including an imaging means, an encoder module, a detection module, a display module, and a central processing unit, (a) converting the image signal photographed using the imaging means into image data in the encoder module; (b) detecting an operation of a subject by receiving the image signal or the image data as an input from the detection module; (c) implementing the operation detected in step (b) on the display module under the control of the CPU; And (d) executing the application using the user interface method implemented in step (c) Application driving method in a camera-equipped mobile communication terminal having a photographing function comprising an image pickup means, an encoder module, a detection module, a display module and a central processing unit. The method of claim 9, And said application comprises at least one of a computer game, a mobile game, a document processor, a spreadsheet, a browser, and a database. A user interface system using an imaging signal of a mobile communication terminal with a camera having a photographing function, Imaging means for photographing a subject to generate an image signal; An encoder module for converting the captured image signal into image data; A detection module configured to detect an operation of a subject by receiving the image signal or the image data as an input; A display module displaying an operation of the subject detected by the detection module; And A CPU for controlling to implement the operation of the subject detected by the detection module on the display module And a user interface system using an image pickup signal of a mobile communication terminal with a camera having a photographing function. The method of claim 10, The detection module is a user interface system using an image pickup signal of a mobile terminal with a camera having a shooting function, characterized in that for detecting the operation of the user's hand or finger. The method of claim 11, The central processing unit attaches a camera having a photographing function, which detects a motion of the hand or a finger and converts the detected motion into a motion of a pointer on the display module to control movement or selection of the pointer. User interface system using the image pickup signal of the mobile communication terminal. The method of claim 10, The detection module calculates a position coordinate to which the pointer should move from a previous pointer position and provides it to the CPU as an input, or the CPU directly calculates a position coordinate to which the pointer should be moved. A user interface system using an image pickup signal of a mobile communication terminal with a camera having a shooting function, characterized in that the position is updated. The method according to claim 12 or 13, If it is detected by the detection module that the subject approaches the image pickup means in the vertical direction, it is determined as the selection, and the selection operation set in the icon indicated by the pointer on the display module is determined as the selection operation. A user interface system using an image pickup signal of a mobile communication terminal with a camera having a shooting function, characterized in that the execution. The method according to claim 12 or 13, Pressing any one button designated by the key input unit of the mobile communication terminal recognizes the selection and executes the selection operation set on the icon indicated by the pointer on the display module. A user interface system using an imaging signal of a mobile communication terminal with a camera having a shooting function. The method of claim 10, The encoder module and / or the detection module may be configured to capture an imaging signal of a camera-equipped mobile communication terminal having a shooting function, characterized in that it comprises at least one of independent hardware, software implemented on independent hardware, and software implemented in a central processing unit. User interface system. The method of claim 10, The display module may include at least one of a super twisted nematic liquid crystal display (STN LCD), a thin film transistor liquid crystal display (TFT LCD), a plasma display panel (PDP), and an organic electroluminescence display (ELD). A user interface system using an image pickup signal of a mobile communication terminal with a camera having a function.