KR20060104607A - Mobile terminal with user interface function using camcoder and user interface method - Google Patents

Abstract

The present invention relates to a mobile phone having a user interface function using a camcorder function and a user interface method. When a button window on which a button is formed is displayed on a screen, the area information of the button on the screen and the RGB value of the button area are grasped; Shooting the indicator medium with a camcorder function to make a digital image, the image of the indicator medium and the image of the button window is mixed with the color of the liquid crystal cell to be displayed on the screen to create a superimposed image, the button area in the superimposed image Detects an overlapping RGB value corresponding to and comparing the overlapping RGB value with the RGB value of the button area in the button window, and when the overlapping RGB value and the RGB value of the button area in the button area are different, the button color change is referred to. And determines an operation corresponding to the button on.

Virtual Touch, Color Data, Overlay, Camcorder

Description

Mobile terminal with user interface function using camcoder and user interface method

1 is a schematic diagram illustrating the concept of the present invention.

Figure 2 is a block diagram of a mobile phone having a user interface function using a camcorder function according to an embodiment of the present invention.

3 is a flowchart illustrating a user interface method using a camcorder function according to an embodiment of the present invention.

4 is an exemplary view showing a button arrangement of a button window according to the present invention.

The present invention relates to a mobile phone having a camcorder function, and more particularly, to capture a movement of a hand or a specific object on a background screen on which a menu button or an execution button is displayed through a camcorder function to display a background screen. The present invention relates to a mobile phone and a user interface method for judging button execution by displaying overlapping and detecting a change in color due to overlap of a menu or an action button with a hand or a specific object.

Commonly known user interface methods include a method of directly operating a keypad, a method of pressing an icon or a button displayed on a touch screen, and a method of inputting a voice and recognizing a voice to determine a user command.

However, the method of using a keypad has a problem that a space constraint occurs when designing a product because a keypad device must be installed in a terminal, and the method of using a touch screen has a problem that a unit price of a product increases because an expensive touch screen is used. The speech recognition method has a problem in that a large amount of process throughput causes a process to be inconvenient and inaccurate speech recognition.

Therefore, the user is easy to operate for users with various needs, can be interesting when operating, and a low cost user interface method is required.

Summary of the Invention An object of the present invention is to solve a conventional problem, and in general, a camera of a user of a camcode function, which visualizes the movement of a user's hand or a specific object and superimposes it on a background screen on which a menu button (or icon) or an execution button is displayed. The present invention provides a mobile phone and a user interface method having a user interface function using a camcorder function that determines that an image of a hand or a specific object is superimposed on a button (virtual touch).

The present invention for achieving the above object, in the mobile phone having a camcorder function, the liquid crystal monitor; A button color information storage unit storing button area information and setting color range information and / or initial color information of the button area in accordance with identification information of each screen window; A control unit which controls the overall operation of the mobile phone and outputs identification information of the screen window displayed on the screen; An application program unit which drives under the control of the controller and outputs screen information for a specific operation; An image processing unit for converting an analog signal received from an image pickup device into a digital signal, processing a frame, and outputting the image data; The screen information provided by the application program unit or the image corresponding to the image data provided by the image processing unit is controlled to be displayed on the liquid crystal monitor, and the event of overlapping the image data on the screen information A screen output unit which calculates and mixes RGB values and RGB values of the image data and displays the color-mixed image data on the liquid crystal monitor; And receiving identification information of the current screen window output from the controller and substituting the identification information into the button color information storage unit to determine first color information of the corresponding button area and the button area, and receiving the mixed image data from the screen output unit. And a button change detector configured to detect a color change by comparing the mixed color information of the button area and the first color information when receiving the color change detection signal. The control unit receives a color change detection signal from the button change detector as a user input. A mobile phone having a user interface function using a camcorder function, which recognizes and performs an operation corresponding to on of a button whose color has been changed.

According to another aspect of the present invention, there is provided a user interface method using a camcorder function of a mobile phone, wherein when a button window on which a button is formed is displayed on a screen, the button area on the screen and the RGB value of the button area are displayed. A first step of identifying; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with an RGB value of the button area in the button window; And a sixth step of determining a button color change when the overlapping RGB value and the RGB value of the button area in the button area are different, and performing an operation corresponding to button on. to provide.

On the other hand, the present invention according to another aspect for achieving the above object is, in the user interface method using the camcorder function of the mobile phone, when the button window on which the button is formed is displayed on the screen, the first to grasp the button area on the screen step; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with a set color range set for detecting a button color change; And a sixth step of determining that the button color is changed when the overlapping RGB value is included in the set color range, and performing an operation corresponding to button on.

The present invention further provides a computer readable recording medium having recorded thereon a program for executing the method.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, detailed descriptions of well-known functions or configurations will be omitted if it is determined that the detailed description of the present invention may unnecessarily obscure the subject matter of the present invention. Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

First, the concept of the present invention will be described with reference to FIG. 1 is a schematic diagram illustrating the concept of the present invention.

When a user request occurs or a default event is started, a background button 90 displays a specific button, for example, a call button B1. In this state, the user places one finger on the screen window 90 while looking at the screen window 90, photographs the finger with the camera of the mobile phone, and displays the captured finger on the background screen window 90. In the screen window 90, both the call button B1 and the finger are displayed as shown in FIG. Since the finger is photographed on the screen, the finger does not need to be placed on the screen. In this regard, it can be seen that the present invention does not matter where the camera lens is located.

As shown in (a) of FIG. 1, when the finger and the call button B1 do not overlap, the initial color of the call button is displayed on the screen, and the finger is also displayed on the screen in the color of the initial color.

In this state, when the user moves the finger in the direction of the call button B1 while viewing the screen and displays the movement of the finger on the background screen window 90 by photographing the camcorder function, the screen window 90 is shown in FIG. ), An image in which the finger and the call button B1 are superimposed is displayed on the screen.

The area of the call button B1 superimposed with the finger is displayed as a color (in this case, light flesh color) mixed with the flesh color of the finger, not the initial white color of the initial call button B1 index.

As a result, when the finger virtually touches the call button B1, the color of the call button B1 area appears.

The present invention provides a user interface for determining whether or not a virtual touch by detecting the color change of the call button (B1) area.

Hereinafter, a configuration of a mobile phone implementing the above-described concept of the present invention will be described with reference to FIG. 2. 2 is a block diagram of a mobile phone having a user interface function using a camcorder function according to an embodiment of the present invention.

As shown in FIG. 2, the mobile phone of the present invention stores the lens unit 10, the image pickup device 20, the image processing unit 30, the screen output unit 40, the button change detection unit 50, and the button color information. The unit 60 includes a control unit 70, an application program unit 80, and a liquid crystal monitor 90.

The lens unit 10 includes one or more lenses and forms light reflecting off a subject. The image pickup device 20 forms an image formed by the lens unit 10, and converts an optical signal for the image into an electrical signal and outputs it to the image processing unit 30.

The image processor 30 converts an analog signal received from the image pickup device 20 into a digital signal, processes a frame, and outputs the image data corresponding to the liquid crystal monitor 90 to the screen output unit 40. do. At this time, the image data processed by the image processing unit 30 is composed of RGB values to be expressed in each cell (combination of RGB) of the liquid crystal monitor 90.

The screen output unit 40 outputs a liquid crystal control signal so that the image data provided from the control unit 70 or the image processing unit 30 is output as an image on the screen of the liquid crystal monitor 90, and the control unit 70 or the image processing unit It has a function of temporarily storing the image data provided by 30. The screen output unit 40 processes an operation of superimposing the image data received from the image processing unit and the image data provided by the control unit 70 by driving an application program during the service of the present invention. Color information (RGB value) expressed in each cell of the liquid crystal monitor 90 is obtained, and color information (RGB value) of each cell due to the superimposition is provided to the button change detection unit 50 and the liquid crystal monitor 90. .

Here, the operation of superimposing the image data uses an RGB value representing color. That is, the bit data of the RGB values of the two colors to be superimposed is calculated to obtain the RGB values of the mixed colors.

The button change detection unit 50 is configured to display the color information (or RGB value) of the button area displayed on the background screen window and the button area overlapped with the screen received from the screen output unit 40 (the image captured by the camcorder function) is reflected. By comparing the color information, it detects the color change of the button area.

There are two methods for detecting the color change of the button area.

First, there is a method of detecting the change of the color information (RGB value) by simply comparing the color information (RGB value) of the initial button area and the color information (RGB value) of the button area after performing the camcorder function. Secondly, the color of the color of the hand (or pointer) and the color of the initial button area while the object pressing the button area is fixed to a hand or a specific object (ie, a pointer) is referred to as 'mixed color'. There is a method of storing in advance and detecting a change in color information by determining whether the color information of the button area is mixed color information after performing a camcorder function.

In the second case, the color of the finger is different for each person, and even if the same color is applied to the pointer, there is a slight difference.In the above, since the RGB color data reflects not only the color information but also the contrast information, it is preferable to set the mixed color information as the range of RGB values. desirable. Although it can be applied to both the first and second methods, it is preferable to apply the conventional edge detection image processing technique of extracting only the edge portion of the object from the image of the finger or pointer captured in the first method.

The button color information storage unit 60 stores the color information of the button area currently displayed in the background screen window, or stores the set mixed color information of the button area after shooting the camcorder, and changes the color information of the button area to the button change. The sensing unit 50 is provided.

The control unit 70 controls the operation of each component, and in particular, provides the button change detection unit 50 with identification information of the screen window displayed on the screen according to the driving of the application program, and the virtual touch event in the button change detection unit 50. When the signal is received, the set operation corresponding thereto is performed.

The application program unit 80 is composed of programs that perform various service operations corresponding to button input. The liquid crystal monitor 90 displays an image on the screen according to the image data output from the screen output unit 40.

Hereinafter, a second method of the above-described color change detection method will be described with reference to FIG. 3. Since the first method of the color change detection method can be easily understood through the second method described below, it will not be described further.

3 is a flowchart illustrating a user interface method using a camcorder function according to an embodiment of the present invention.

By the operation by the keypad and / or the default operation according to the power-on and / or the virtual touch operation of the present invention, the button window on which the button (or icon) is arranged is displayed on the screen of the liquid crystal monitor 90 (S301).

When the button window is displayed, the controller 70 grasps the identification information of the button window displayed on the current screen and the area information (coordinate information) or / and the color information of each button area arranged on the button window, and then changes the button change detection unit ( 50) (S302). The identification information of the button window and the area information of each button are fixed values.

In this state, when the user instructs to shoot the camcorder, the camcorder function is started (S303). Accordingly, when the user places the hand or the pointer in front of the lens unit 10, the lens unit 10 forms the light reflected by the hand or the pointer on the image pickup device 20, and the image pickup device 20 forms an image. Provides an electrical signal to the image processor 30.

When the image processor 30 receives an electrical signal on the subject, the image processor 30 converts the electrical signal into a digital signal and then performs image processing and / or image processing (S304). By the image processing of the image processing unit 30, the subject image formed on the lens unit 10 becomes image data for one frame, and / or the finger edge portion of the image data by image processing (edge detection image processing) or Only the image of the edge portion of the pointer remains (S305).

The image or / and image processed data by the image processing unit 30 is provided to the screen output unit 40, and the screen output unit 40 uses an RGB value to superimpose the received image data with the currently displayed image data. The color mixing operation is performed, and image data resulting from the color mixing is provided to the button change detection unit 50 (S306).

Then, the button change detection unit 50 extracts the RGB values corresponding to the button area obtained in step S302 from the image data of the received color mixing result to determine the RGB value of the button area or the color of the button area (S307).

Here, the RGB value grasped in step S307 is not a single RGB value corresponding to one color, but a range of RGB values with color change among all RGB values identified as button areas. The range of RGB values is generally that the object does not produce the same RGB values because the contrast is generated by the outside or itself. Therefore, the RGB value grasped in step S306 is obtained in a specific range.

On the other hand, the user preferably overlaps the entire button area, but only a part of the button area can be overlapped with a finger or a pointer. Therefore, the button change detection unit 50 of the present invention preferably performs color change detection only when the overlap area is greater than or equal to the set area in the button area.

When the button change detection unit 50 grasps the RGB value of the button area from the superimposed image data, the button change detection unit 50 provides identification information (code or number) of the button window obtained from the control unit 80 to the button identification information storage unit 60. The set color range information matched to the button identification information is obtained, and the set color range information is compared with the RGB value of the button area after the superimposition (S308).

In the comparison, if the RGB value of the button area after overlapping is not included in the set color range, the button change detection unit 50 determines that there is no color change (S309), and continues the current operation. However, the button change detection unit 50 determines that there is a color change when the RGB value of the button area after overlapping is included in the set color range in the comparison, and determines that the button is input accordingly (S310).

In addition, the button change detection unit 50 provides a signal for notifying the button input to the control unit 70, and the control unit 70 performs an operation corresponding to the button input (S311).

Meanwhile, in order to recognize an intentional touch of a user, the present invention may set a color change detection time, and determine that the touch is a virtual touch when the time when the color change is detected exceeds the set time.

In order to achieve the above-described embodiments of the present invention, it is advantageous to have fewer buttons arranged on the screen, and to avoid touching an unintentional button area when moving a finger or a pointer.

Therefore, the arrangement of the buttons arranged on the screen of the present invention is preferably arranged as shown in FIG. 4 is an exemplary view showing a button arrangement of a button window according to the present invention.

Referring to FIG. 4, the plurality of buttons A-I are disposed at the edges of the screen and have no arrangement in the center of the screen. In this way, the upper buttons (A, I, H) can be selected by moving a finger or a pointer from the top down, and the left buttons (A, B, C) are selected by moving a finger or a pointer from left to right. Just do it. The right buttons H, G and F may be selected by moving a finger or a pointer from right to left, and the lower buttons D and E may be selected by moving a finger from bottom to top.

The technical spirit of the present invention has been described above with reference to the accompanying drawings, but this is by way of example only and not intended to limit the present invention. In addition, it is obvious that any person skilled in the art can make various modifications and imitations without departing from the scope of the technical idea of the present invention.

The present invention provides a user interface of a new method that can cause an interest in use, such as using a touch screen without having a separate device in the camcorder function, and does not have to be spatially restricted for the user interface.

Claims (12)

On mobile phones with camcorder function, A liquid crystal monitor; A button color information storage unit storing button area information and setting color range information and / or initial color information of the button area in accordance with identification information of each screen window; A control unit which controls the overall operation of the mobile phone and outputs identification information of the screen window displayed on the screen; An application program unit which drives under the control of the controller and outputs screen information for a specific operation; An image processing unit for converting an analog signal received from an image pickup device into a digital signal, processing a frame, and outputting the image data; The screen information provided by the application program unit or the image corresponding to the image data provided by the image processing unit is controlled to be displayed on the liquid crystal monitor, and the event of overlapping the image data on the screen information A screen output unit which calculates and mixes RGB values and RGB values of the image data and displays the color-mixed image data on the liquid crystal monitor; And Receives the identification information of the current screen window output from the control unit and substitutes it into the button color information storage unit to grasp the first color information of the button area and the button area, and receive the mixed image data from the screen output unit. And a button change detector for detecting a color change by comparing the mixed color information of the button area with the first color information. The control unit recognizes the color change detection signal from the button change detection unit as a user input and performs an operation corresponding to the on of the button with the color change, characterized in that the mobile phone with a user interface function using a camcorder function. The method of claim 1, The first color information is a mobile phone having a user interface function using a camcorder function, characterized in that the set color range information or the initial color information of the button area. The method according to claim 1 or 2, The button change detection unit is a mobile phone having a user interface function using a camcorder function, characterized in that for determining that the color change of the button area lasts more than a predetermined time. The method of claim 3, The image processing unit detects only an edge portion of a finger or a pointer by processing an edge detection image on image data for each frame, and then provides only image data corresponding thereto to the screen output unit. Mobile phone with a. The method of claim 1, And the button area is located at the edge of the screen. In the user interface method using the camcorder function of the mobile phone, A first step of grasping the area information of the button on the screen and the RGB value of the button area when the button window on which the button is formed is displayed on the screen; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with an RGB value of the button area in the button window; And And determining a button color change if the overlapping RGB value and the RGB value of the button area in the button area are different, and performing an operation corresponding to button on. In the user interface method using the camcorder function of the mobile phone, A first step of identifying area information of the button on the screen when the button window on which the button is formed is displayed on the screen; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with a set color range set for detecting a button color change; And And determining a button color change when the overlapping RGB value is included in the set color range, and performing an operation corresponding to button on. 6. The method of claim 7, wherein And the set color range is a range of RGB values for a mixed color obtained by mixing the initial color of the button area and the color of the indicator medium. The method of claim 6, The method may further include extracting a superimposed image only at the end of the indicator medium by performing edge detection image processing on the superimposed image after the superimposed image is generated by the third step. User interface method used. The method according to claim 8 or 9, In the sixth step, the duration of the button color change is counted, and the counter duration is compared with the set duration for detecting the color change duration, and when the counter duration exceeds the set duration, the button is changed. The user interface method using the camcorder function of the mobile phone further comprising the step of determining that the on. On your phone, A first step of grasping the button area and the RGB value of the button area on the screen when the button window on which the button is formed is displayed on the screen; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with an RGB value of the button area in the button window; And The computer-readable recording in which the program for executing the sixth step of performing the operation corresponding to the button-on is determined if the overlapping RGB value and the RGB value of the button area in the button area are different. media. On your phone, A first step of identifying the button area on the screen when a button window on which a button is formed is displayed on the screen; A second step of photographing the indicating medium with a camcorder function and making the digital image; Creating a superimposed image by color-mixing an image of the indicator medium and an image of the button window based on a liquid crystal cell to be displayed on a screen; A fourth step of detecting an overlapping RGB value corresponding to the button area in the overlapping image; A fifth step of comparing the overlapping RGB value with a set color range set for detecting a button color change; And And determining that the button color is changed when the overlapping RGB value is included in the set color range, and executing a sixth step of performing an operation corresponding to button on.

Images