JP2007527046A - Operation area selection in a computer device with a graphical user interface - Google Patents

Abstract

An operation area (for example, key or button) represented by a color map stored on the device in the GUI of the computer device based on the touch screen. The color map has a region where the shape and size completely correspond to each of the operation keys or buttons. Each region is completely filled with unique colors. In use, if an XY coordinate set of a particular selected / touched point is generated, it is easy and very fast to obtain the color of a pixel that exists at the same XY coordinate in the colormap . be able to. For this reason, it is very fast to determine the operation area associated with the color of the pixel by matching in the table storing each operation area together with the color associated with it on the color map. It can be done easily.

Description

  The present invention relates to a method for selecting an operation area in a computer device having a graphical user interface, and more particularly to a device having a touch screen.

  One common problem in operating systems based on Symbian OS and other GUIs (graphical user interfaces) for computer devices is which operating area within the GUI was actually selected by the user, for example Determining which particular operation key or button will be selected when the user taps the touch screen. When the stylus taps the display screen, the contact position is converted into XY coordinates. Touch screens typically include an ITO layer on a display (such as an LCD) to detect pressure. When pressure is applied to the ITO overlay, the top soft surface is locally deformed and the deformed area contacts a rigid solid substrate. The level of the analog voltage across the ITO coating on the internal surface layer is then interpreted by the device, giving the XY coordinates of the pressure position.

  Next, the XY coordinates are given to the current window corresponding to the coordinates by the OS. The window has a plurality of operating areas, and each area will typically be queried to find out which corresponds to the coordinates.

  FIG. 1 is a diagram illustrating a design of an operation UI implemented to enable user input. In FIG. 1, the operation UI defines seven operation areas that can be selected by the user. These are up / down / left / right / confirmed / Yes / No.

  In order to express these operation areas on the OS, each area is defined by a rectangle. For example, as shown in FIG. 2, the “up” operation UI will be recorded as a rectangle with the upper left coordinate set as X68, Y4 and the lower right coordinate set as X111, Y29. FIG. 2 shows all seven rectangles for seven different operation areas.

Since each of the seven selectable operating areas is roughly defined as a rectangle, it is possible to determine which area was selected with the pen / stylus. When the user touches the display screen on the given operation area, XY coordinates at the position where the pen touches are generated. Next, the device evaluates each rectangle to see if the XY coordinates fall within the rectangle boundary. One common algorithm for determining whether a set of XY coordinates falls within a rectangular boundary is as follows.
1. Is the X coordinate selected by the user larger than the upper left X coordinate of the operation area?
2. Is the X coordinate selected by the user smaller than the lower right X coordinate of the operation area?
3. Is the Y coordinate selected by the user larger than the upper left Y coordinate of the operation area?
4). Is the Y coordinate selected by the user smaller than the lower right Y coordinate of the operation area?
If all the above conditions are true, we can know that the user has selected that operating area.

  In order to find the operation area selected by the user, it is necessary to execute this algorithm for all rectangles. If many of the operation UI designs are complex and there are many rectangles, this can be a relatively time consuming process.

  Another drawback is that the part of the operating area that can be captured is only a rectangle. This means that a design of an operation UI with a concentrated figure, such as an operation button having a complicated shape, may make it difficult to accurately capture an instruction event. FIG. 3 shows an area where the user has selected the “down” operation. This event will not be detected because it is outside the "bottom" rectangular boundary.

Although it is possible to capture user input by using a rectangle, there are three main drawbacks.
• The hit area must be rectangular. An operation UI having an arbitrary shape cannot be accurately selected.
The algorithm that evaluates the two sets of coordinates must be run for all hit regions.
It takes time to change the design of the operation UI because all the coordinate pairs describing the hit rectangle must be recalculated.

In a first aspect, a method for determining an operation area selected by a user from among operation areas displayed on a display of a computer device is such that each of a plurality of different selectable operation areas is stored in a memory of the apparatus. Is associated with one unique color of a set of unique colors in a color mask that is stored and not displayed on the display, the color mask corresponding to each of the operational areas and the unique color Each of which is colored into one of the colors
(A) generating a set of coordinates relating to a contact position on the display;
(B) obtaining a color of the color mask at the set of coordinates;
(C) determining the operation region associated with the same color as the acquired color;
Is provided.

  For this reason, instead of performing rough rectangle approximation for the operation keys and buttons, the present invention generates a color mask having an area whose shape and size preferably correspond exactly to the operation keys and buttons, and stores them in the memory. It needs to be stored. The area is completely filled with a unique color. All other operation keys or buttons displayed at the same time are given corresponding unique color areas in the color mask. Thus, the device will store a different set of color masks, each color mask corresponding to a different operation key / button arrangement. Some color masks may be baked into the device's ROM (for example, for operational GUI designs that are intended to be used at all times). Alternatively, other color masks may become part of the application software that is loaded into the device by the end user. However, the color mask relates to the design of the operation GUI unique to the application software. The color mask is not displayed on the screen at all, so it is completely “virtual”.

  In use, if a set of XY coordinates for a particular selected / touched point is generated, it is easy and very fast to obtain the color of the pixel present at the same XY coordinate in the appropriate color mask. It can be carried out. For this reason, the operation region corresponding to the color can be determined very easily and easily. For example, consider an operation UI design in which an icon is displayed in the center of the screen. It is assumed that the icon has a complicated shape such as a flower. When that flower is selected, the image application is opened. It is assumed that there are no other operation areas on the same screen. In this case, when a device or image application is specified, a color mask having exactly the same shape as the flower is generated. In the mask, one color, for example blue, is given inside the flower. The rest of the color mask is black. The color mask is stored in the memory of the device. It is not displayed on the screen at all.

  When the device is in use, a flower icon is displayed. The user contacts the desired part of the flower. Contact coordinates are generated and used to obtain the color at the same coordinates of the color mask. When the color of the color mask is blue, the apparatus uses a table that associates each color with the operation area (for example, blue is a flower, black is not functioning) and the color s (blue And black). And it can determine at high speed that the flower was selected. If the color mask is black, the device knows that the user has touched the screen outside the flower icon. If the design of the operational UI is complex, evaluating all the different rectangles using prior art techniques will be complex and time consuming. With the color mask approach of the present invention, we only compare different color regions. This is fast and computationally efficient. In particular, this is an important advantage in a computer device with limited resources, such as a high-function telephone or another type of cellular phone.

  The term “computer device” includes all types of computer devices that are operated by a human by selecting an operation area displayed on a display. For example, the term includes a dedicated device such as a PDA, a mobile phone, and a digital audio player, a remote control unit, and the like. The term “operation area” should be interpreted extensively to include any object that the user will “select” in some way (where “selection” is, for example, (a) text data / Data input such as numeric data, (b) Start of an action, (c) Control or operation of a function, or (d) Selection for specifying data to be operated). Accordingly, the term “operation area” includes keyboard keys or tiles (eg, letter or number keys), operation buttons or sliders, jog dials, joysticks, or roller buttons. The term “operation area” includes items that are not themselves objects. In particular, any icon or image that the user will select is included.

  Embodiments according to the present invention will be described below in detail with reference to the accompanying drawings.

  The present invention is described by reference to an implementation that can be executed on a Symbian OS provided by Symbian, London, England. The Symbian OS is designed for high function phones and other mobile phones. The limited processing power present in this type of device makes the present invention particularly valuable.

(Solution by color mask)
According to the color mask solution, a plurality of rectangular hit area techniques are replaced with a color mask stored in a memory. A color mask is a direct representation of the operating area where we are trying to get input, with a different area for each of the different operating areas. To create a mask, simply obtain the original bitmap for the operation region and fill the corresponding color mask region with the same color that we need to obtain user input.

  FIG. 4 shows a color mask showing the operation UI of FIG. Color mask areas that are not considered user input are colored in black. Then, for example, in FIG. 4, the color mask region of interest can be marked with a different color such that red is confirmed, yellow is left, blue is right, and purple is bottom.

  In Symbian OS, each color is stored as a 32-bit unsigned integer in the format of BB, GG, RR (blue, green, red), and therefore, basic red is 0x0000ff, green is 0x00ff00, and blue is 0xff0000.

As shown in Table 1 below, a table of known colors for a region as a member variable associated with a reference to the corresponding operation region is stored in the memory of the device.
[Table 1]
Const TUint KYesAreaColour = 0xf0ff05; // Cyan
Const TUint KNoAreaColour = 0x80c5f6; // Beige
Const TUint KUpAreaColour = 0x00ff00; // Green
Const TUint KDownAreaColour = 0xf773d0; // purple
Const TUint KLeftAreaColour = 0x03faf7; // Yellow
Const TUint KRightAreaColour = 0xff0000; // Blue
Const TUint KConfirmAreaColour = 0x0000ff; // Red
Const TUInt KblackIgnoreArea = 0x000000; // Black The color mask is also stored in the device memory as a member variable.

  The usage of the color mask is as follows. If the user taps with the stylus and selects the “left” operating area, the application will receive an event related to coordinates X58, Y41 that indicate the touch position of the stylus.

  Next, the program related to the selected operation area inquires about the pixels at the positions X58 and Y41 of the color mask. As described above, the color mask is not actually displayed on the display and exists only on the memory. The color arranged in the color mask has nothing to do with the color displayed on the display.

  In SymbianOS, all bitmaps are represented in memory as a collection of pixels (a pixel is an unsigned integer representing a color, in the same format as storing a known color). SymbianOS and other GUI-based operating systems provide a fast method of obtaining pixels from a bitmap in memory, and this feature is also used to obtain X58, Y41 pixels.

この場合、表１に示すように、ピクセル値0x03faf7はKLeftAreaColour値（0x03faf7）に対応するだろう（即ち、プログラムは、ピクセルの色の黄を、その色に対応する操作領域、つまり「左」の操作領域に対応させる）。従って、プログラムはユーザが「左」を選択したと決定するであろう。プログラム命令文の対応する部分は太字で示している。ポイントとなる工程を図６に要約する。 In this case, the X58, Y41 pixels would be obtained from the color mask as TUint 0x03faf7 (corresponding to yellow). The program would then use the following statement:

In this case, as shown in Table 1, the pixel value 0x03faf7 will correspond to the KLeftAreaColour value (0x03faf7) (ie the program will change the yellow color of the pixel to the operation area corresponding to that color, ie "left" Corresponding to the operation area). Thus, the program will determine that the user has selected “left”. Corresponding parts of program statements are shown in bold. The key steps are summarized in FIG.

  Any combination of colors (including grayscale) can be used provided each region of the color mask has a unique value.

  Since we compare a single unsigned integer with the integers associated with each of the different regions (corresponding to the operation region one-to-one), this approach is very efficient in terms of processing time. If the design of the operational UI is complex, evaluating all the different rectangles using the prior art approach will be complex and time consuming. According to the color mask technique, different color regions need only be compared.

(Customize)
Another advantage of the color mask approach is that the appearance of the operating UI design can be completely different by simply updating the mask and the original operating UI bitmap.

  FIG. 7 shows a completely new design of the original operating UI (of FIG. 1). However, this new design still has seven operating areas that are exactly the same as the original operating design, eg, top / bottom / left / right. This new design contains a lot of non-rectangular hit areas, so it would be extremely difficult to represent with a rectangular approach.

  Updating this operational design with the color mask approach simply requires generating a new color mask bitmap representing the new operational design. A new color mask is shown in FIG. 8, which can be generated by anyone with little technical knowledge of the drawing application. The only criterion is that the same color value is again used, for example, 0x03faf7 (yellow) in the operating area of the “left” arrow. If new color bitmaps and color masks are added to the application, no program code modifications are required. This is because the color mask technique only needs to know which color indicates which operation action (eg, as shown in Table 1).

It is a figure which shows the design of the operation GUI which has a 7 different operation area | region displayed on a part of screen of a computer apparatus. It is a figure which shows typically the conventional method based on a rectangle for determining the operation area selected by the user among seven operation areas. It is a figure which shows that a complicated operation area | region cannot be processed successfully according to a prior art. It is a figure which shows the color mask used in this invention. There are seven different operation areas, each of which is exactly equal in shape and size to the corresponding operation area of the design of the operation GUI displayed on the display. It is a figure which shows a mode that the apparatus stores the bit table of the original operation UI design, the color mask, and the table of the color linked | related with the operation area | region. It is a figure which shows the flowchart of the basic process in this invention. It is a figure which shows the design of a different operation GUI which has the same kind of operation area as the design of the operation GUI of FIG. The design of this different operation GUI is a diagram showing how it can be realized simply using different color masks.

Claims (17)

A method for determining an operation region selected by a user from among operation regions displayed on a display of a computer device, wherein each of a plurality of different selectable operation regions is stored in a memory of the device and stored in the display. Is associated with one unique color of the set of unique colors in the color mask that is not displayed, the color mask corresponding to one of the operating areas and one of the unique colors. Each area is colored,
(A) generating a set of coordinates relating to a contact position on the display;
(B) obtaining a color of the color mask at the set of coordinates;
(C) determining the operation region associated with the same color as the acquired color;
A method comprising the steps of: The method of claim 1, wherein the color mask is obtained using a bitmap of the operation area. 3. The method of claim 2, wherein a table associated with the set of unique colors is stored in a memory of the device, with references to selectable operation areas to which the unique colors are respectively associated. The method of claim 3, wherein each of the unique colors in the table is represented as an unsigned integer. Each of the unique colors in the color mask is represented as an unsigned integer, and the unsigned integer representing the color in the set of coordinates is an unsigned integer in the table until a match is found. The method of claim 4, wherein the method is compared with each of the following. 6. The method of claim 5, wherein if a match is found, a corresponding selectable operating area is determined using the table. The method according to claim 1, wherein a selectable operation region can have the arbitrary shape as long as the region corresponding to the arbitrary shape in the color mask can be filled with a single color. The arrangement and design of the different selectable operation areas can be updated to different arrangements and designs by changing the bitmap of the operation area and the color mask. the method of. The method according to claim 8, wherein the bitmap of the operation area and the color mask is changed using a drawing application. A computer device adapted to determine an operation region selected by a user from among operation regions displayed on the display of the device, wherein each of a plurality of different selectable operation regions is stored in a memory of the device. Associated with one of the sets of unique colors in the color mask that are stored in the display and are not displayed on the display, the color masks corresponding to one of the operating areas, and Each of the regions is colored one of the unique colors, the device
(A) generating a set of coordinates relating to a contact position on the display;
(B) obtaining a color of the color mask at the set of coordinates;
(C) The computer apparatus is adapted to determine the operation area associated with the same color as the acquired color. The apparatus according to claim 10, wherein the color mask is obtained using a bitmap of the operation area. 12. The apparatus of claim 11, wherein a table associated with the set of unique colors is stored in a memory of the apparatus along with references to selectable operation areas to which the unique colors are associated. The apparatus of claim 12, wherein each of the unique colors in the table is represented as an unsigned integer. Each of the unique colors in the color mask is represented as an unsigned integer, and the unsigned integer representing the color in the set of coordinates is an unsigned integer in the table until a match is found. 14. The device according to claim 13, characterized in that it is compared with each of the following. 15. The apparatus of claim 14, wherein if a match is found, a corresponding selectable operating area is determined using the table. The apparatus according to claim 10, wherein a selectable operation region can have the arbitrary shape as long as a region corresponding to the arbitrary shape in the color mask can be filled with a single color. Application software programmed to run on a computer device,
Displaying each of a plurality of different selectable operating areas on the device;
With a color mask,
Each of the color masks corresponds to one of the operation areas, and each area is colored with a unique color.
Application software, wherein the color mask is stored in a memory of a device and is not displayed on the display.

Images