KR20050094037A - Image control - Google Patents

Abstract

The present invention relates to the field of hand-held devices that are equipped wit a processor and a digital camera for capturing motion video or still images. Images captured by the camera are used to determine motion of the hand-held device. A resulting motion signal is used as input to a user interface. Displayed images can be scrolled, zoomed or rotated by moving the hand- held device. The motion signal can also be used as input for a graphical user interface to move a cursor or other object of the graphical user interface over the display. The invention relates further to a hand-held device provided with means for sensing motion, a display, a keypad with at least a first- and a second key, and a graphical user interface with objects and a cursor. The hand-held device also comprises means for transforming the sensed motion of the handheld device into a signal suitable for moving the cursor over the display.

Description

Image control

FIELD OF THE INVENTION The present invention relates to the field of hand-held devices equipped with a digital camera and a processor for capturing moving images or still images, and more particularly to devices further comprising a display or a graphic or text based user interface for displaying images. will be.

Hand-held devices with digital image capture equipment, digital processing power and high resolution displays are becoming increasingly common in a wide variety of applications.

For example, small mobile phones have recently been equipped with digital cameras and relatively small high resolution LCD screens. Hand-held computers, commonly referred to as "personal digital assistants" (PDAs), are also available and are typically equipped with small high resolution display screens and with slots for storing digital cameras, for example. Similarly, communication devices with both cellular communication and computer functionality are typically available, with small display screens and a built-in or detachable digital camera. Such small, hand-held devices do not have conventional input devices, such as computer mice and other control keys, and cannot conveniently have. Therefore, conventional personal computer interfaces, which also have their own disadvantages, are not suitable for such small hand-held devices.

As a result, when obtaining output, for example when viewing data on a display screen and when entering commands, for example, changing the area shown on the display screen or controlling the performance of certain parameters associated with the device. Considerable limitations exist when using such small hand-held devices. Moreover, if limited areas are available on the entire device as well as the display screen, adding additional control keys is difficult and cumbersome for a user requiring two-handed operation of the device.

US 6466198 discloses a system and method for view navigation and magnification of a display of hand-held devices in response to orientation changes along only two axes of rotation as measured by sensors inside the hand-held devices. The view navigation system can be started and controlled by simultaneously pressing the switches on both sides of the hand-held device. Miniature sensors, such as accelerometers, tilt sensors or magnetoresistive direction sensors, detect orientation changes. Currently these miniature sensors are typically not standard equipment for hand-held devices. Thus, these sensors add cost, use valuable space and add weight.

The present invention is directed to overcoming one or more of the above problems.

1 shows a preferred embodiment of a hand-held device according to the present invention.

2 shows a block diagram of a hand-held device of the embodiment of FIG. 1.

Figure 3 illustrates the axis of rotation and axis of rotation in which the hand-held device is moved and rotated to generate user input in accordance with the present invention.

Figures 3.1 to 3.7 illustrate the use of the present invention for zooming, scrolling and rotating images displayed on a display.

3.8-3.10 illustrate the use of the present invention for user input to a graphical user interface to scroll a table within a window and to resize the window.

3.11 to 3.13 illustrate the use of the present invention for scrolling an enlarged window on a display.

3.14 and 3.15 illustrate the use of the present invention to move a portion of an image by cropping and pasting.

Figure 3.16 illustrates the use of the present invention through a text editing application.

Figure 3.17 illustrates the use of the present invention through an application for entering notes in a stave.

Figure 3.18 illustrates the use of the present invention through a maze game.

Figure 3.19 illustrates the use of the present invention for controlling a video game.

Figure 3.20 illustrates the use of the present invention through an application for controlling the sound settings of the music player function.

4 schematically illustrates a software flow diagram for zooming, scrolling and rotating images displayed on a display.

5 and 6 show an alternative preferred embodiment of the hand-held device according to the invention.

Against this background, the object of the present invention is a hand of the type referred to earlier, which allows for user input with the same hand holding the device, without the need for dedicated sensor equipment used by prior art hand-held devices. -To provide a handheld device.

The object is a hand-held device comprising a processor, a digital camera for capturing moving images or still images and means for converting signals from the camera into motion signals representing movement of the hand-held device. By providing an apparatus, it is achieved according to claim 1.

Thus, by using a sensor available to start with many hand-held devices, i.e., a digital camera for generating auxiliary signals, i.e., a motion signal representing the movement of the hand-held device, a hand-held device with motion sensing is economical. It is provided in a reliable and reliable way.

The hand-held device may further comprise a user interface used as a user input via a motion signal from which the movement of the hand-held device is obtained.

The hand-held device may further comprise a display, preferably a display suitable for displaying the captured images.

The movement of some type of hand-held device may be used to manipulate images at least partially displayed on the display, preferably by moving the images in a manner substantially corresponding to the movement of the hand-held device.

Different types of movement of the hand-held device can be used, for example, to move, and / or zoom, and / or zoom and / or rotate images displayed on the display.

Movement substantially parallel to the plane of the display of the hand-held device may be used to scroll the image displayed on the display. Movement substantially perpendicular to the plane of the display can be used to zoom the image displayed on the display. Rotation of the hand-held device can be used to rotate the image displayed on the display.

The manipulated images can be, for example, images previously captured by the camera.

The movement of the image can be reversed with respect to the movement of the hand-held device because some users may prefer this.

The user interface may comprise a graphical user interface and the movement of the hand-held device may be used as input to the graphical user interface.

The movement of the hand-held device may preferably be used to manipulate the object represented by the graphical user interface by moving the object in a manner substantially corresponding to the movement or inverted movement of the hand-held device, The object displayed by the graphical user interface may be an icon, dialog box, window, menu or pointer.

Certain types of movement of the hand-held device may be used to move, and / or zoom, and / or zoom and / or rotate the objects represented by the graphical user interface.

Movement substantially parallel to the plane of the display of the hand-held device may be used to scroll the object displayed by the graphical user interface. The movement substantially perpendicular to the plane of the display can be used to zoom the object displayed by the graphical user interface. The rotational movement of the hand-held device can be used to rotate the object displayed by the graphical user interface.

The digital camera may be a built-in camera or may be a detachable camera. The camera is movable relative to the hand-held device.

The means for converting the signal from the camera into a motion signal preferably obtains the motion signal from changes between successive images captured by the camera.

The camera may be equipped with an auto focus system, and the focus setting of the auto focus system may be used to detect movement in the camera direction.

The hand-held device may further comprise at least one key. The function of the movement type may depend on the state of the at least one key.

The rotational movement of the hand-held device about an axis substantially perpendicular to the display is such that the image or object becomes static relative to the display in such a way that the image or object is static relative to a fixed coordinate system where the hand-held device is located. It can be used to cause reverse rotational movement of the image or graphical user interface object.

The motion signal can be used to adjust device settings, such as sound settings, keypad settings, and display settings.

The hand-held device may further comprise a keypad having at least a first key and a second key, the graphical user interface comprising a cursor, the movement of the hand-held device moving the cursor to the graphical user interface. Main functions associated with the related object may be activated by pressing the first key and auxiliary functions associated with the related object may be activated by pressing the second key.

The function of the first key may be associated with the selection and activation of objects of the graphical user interface, and the function of the second key may be associated with invoking a context-sensitive menu.

The selection of the related object may be performed by pressing and releasing the first key. Activation of the related object can be performed by pressing and releasing the first key twice in quick succession. Movement and resizing of the related object may be performed by keeping the first key down while moving the hand-held device to move the cursor.

The first and second keys may be softkeys, the current function of the softkeys being shown in dedicated fields of the display, preferably of the display.

The first key may be disposed below the display, on the left side of the display, preferably near the bottom edge of the display, and the second key is preferably near the bottom edge of the display, on the right side of the display. It may be disposed below the display.

Another object of the present invention is to provide an improved method for providing user input to hand-held devices. The object is achieved by providing a method for generating user input for a hand-held device having a processor, a user interface and a digital camera for capturing moving or still images, the method comprising:

Determining movement of the hand-held device from the camera signal; And

Using the determined movement of the hand-held device as input to the user interface.

Another object of the invention is to provide the use of a digital camera of a hand-held device having a processor for generating a motion signal indicative of the movement of the hand-held device.

Another object of the present invention is to provide a hand-held device with an improved graphical user interface. The object is to provide a processor, means for sensing movement of the hand-held device, a display, a keypad with at least a first key and a second key, a graphical user interface with objects and a cursor, and sensing of the hand-held device. It is achieved by providing a hand-held device comprising means for converting the moved movement into a signal suitable for moving on the display.

By controlling the position of the cursor through the movement of the hand-held device, it becomes possible to provide a user-friendly cursor control graphical user interface for the hand-held device.

Advantageously, movement of the hand-held device is used to position the cursor on objects of the graphical user interface and primary functions associated with the related object are activated by pressing the first key and associated with the associated object. Functions are activated by pressing the second key.

The function of the first key may be associated with the selection and activation of objects of the graphical user interface, and the function of the second key may be preferably associated with invoking a context-sensitive menu.

The selection of the related object is preferably performed by pressing and releasing the first key, and the activation of the related object is preferably performed by pressing and releasing the first key twice in quick succession.

The movement or resizing of the related object may be performed by keeping the first key down while moving the hand-held device to move the cursor and the related object in line with the cursor.

The first key and the second key are softkeys and the current function of the softkeys is shown in dedicated fields of the display, preferably of the display.

The first key may be disposed below the display, on the left side of the display, preferably near the bottom edge of the display, and the second key is preferably near the bottom edge of the display, on the right side of the display. It may be disposed below the display.

Means for converting the movement of the hand-held device into a signal suitable for moving the cursor on the display may further include a tilt sensor and / or an image capture device and / or an accelerometer.

The hand-held device may further comprise means for converting signals from the image capture device, ie the camera and / or tilt sensor and / or accelerometer, into position signals for the cursor.

Preferably means for converting a signal from the camera into a motion signal obtains the motion signal from changes between successive images or portions of successive images captured by the camera.

The camera may have an auto focus system, the focusing setting of the auto focus system being used to detect movement in the camera direction.

The graphical user interface may include one or more object types of icons, dialog boxes, windows, menus, and pointers.

Other objects, features, advantages and attributes of the hand-held device, the use of the digital camera in the hand-held device according to the invention and a method for providing user input will become apparent from the detailed description.

In the following detailed description of the present description, the invention will be described in more detail with reference to exemplary embodiments shown in the drawings.

The present invention allows hand-held communication or computing devices with relatively small displays to receive user input by moving or rotating the device. In particular, with devices with a display, the present invention allows for convenient navigation of large stored virtual displays or objects within a graphical user interface. Such devices include PDA devices, camcorders, digital photo cameras, digital binoculars (solid solid stereoscopic imaging system for viewing and digitally recording magnified stereoscopic images), mobile hand-held terminals, advanced pagers, Mobile phones and communicators.

According to a preferred embodiment, the hand-held device is a hand portable phone, preferably a cellular / mobile phone.

1 shows a mobile phone according to the invention, in which the phone, generally indicated as 1, has a user interface 2 with a keypad, a display 3, an on / off button 4, a speaker 5 (openings) It will be appreciated that only bays are shown) and a microphone 6 (only openings are shown). The telephone 1 according to the preferred embodiment is suitable for communicating over a cellular network, such as a GSM 900/1800 MHz network.

The keypad 7 includes keys of the first group of keys 8 as alphanumeric keys, through which the user can enter a telephone number, write a text message (SMS), (related to the telephone number) ) You can write your name. Each of the twelve alphanumeric keys 8 is provided with the numeral "0-9" or the sign "#" or "*", respectively. In character mode each key is associated with a number of characters and certain symbols used in text editing.

The keypad 7 additionally comprises two softkeys 9, two call processing keys 12 and an arrow key 10. The function of the softkeys depends on the state of the phone and menu navigation can be performed using the navigation-keys. The current function of the softkeys 9 is indicated in the individual fields in the display 3, directly above the keys 9. The two call processing keys 12 are used to set up a call or conference call, call termination or incoming call rejection. The key layout is characteristic of, for example, a Nokia 6210® phone.

The arrow key 10 is an up / down key that can be used for cursor movement and scrolling and is centered on the front of the phone between the display 3 and the group of alphanumeric keys 7. A battery pack 14 is mounted on the back of the phone and powers the electronic elements of the mobile phone.

The phone has a flat panel display 3, typically consisting of an LCD, with an optional backlight, such as a TFT matrix capable of displaying color images.

The phone is equipped with a digital camera 35 in which only the lens 36 is visible in FIG. 1. The camera is arranged on the back of the phone above the battery pack 14. The camera direction is therefore substantially perpendicular to the back of the phone 1. Directly beneath the camera is an infrared port 38 for IrDA communication (only the window of the port is shown). The camera may alternatively have a rotatable connection (not shown) to the phone to allow adjustment of the camera orientation with respect to the housing of the phone.

Figure 2 schematically shows the most important parts of the preferred embodiment of the telephone, in block diagram form. The processor 18 controls communication with the network via the transmitter / receiver circuit 19 and the internal antenna 20. The microphone 6 converts the user's voice into analog signals, the analog signals formed by the A / D converter in an A / D converter (not shown) before the voice is encoded in the digital processing unit 14 (DSP). do. The encoded speech signal is transmitted to a processor 18, which supports GSM terminal software. The processor 18 also includes a RAM memory 17a and flash ROM memory 17b (as well as data, power, etc.), a SIM card 16 and a keypad 2, for peripheral units of the device. Form an interface. The digital signal-processing unit 14 voice decodes the signal, and the signal is transmitted from the processor 18 to the earphone 5 via a D / A converter (not shown). The processor communicates bidirectionally with an IrDA port 38 (infrared port) that allows data communication with other ports-equipped devices, such as PCs, laptops, personal digital assistants (PDAs), and other mobile phones. . The phone may be further equipped with a short range RF transmitter receiver (not shown) for data transmission with other devices as mentioned for IR data communication, for example according to the Bluetooth standard.

The processor 18 also forms an interface to the display controller 31. Image controller 31 receives image files from the processor 18 and renders images for display on the display 3.

Camera controller 37 is also connected to the processor 18 and controls the digital camera 35. The camera controller 37 sets the resolution, refresh rate, focus and zoom magnification of the camera 35. The camera controller 37 automatically sets the focus through any of the well-known autofocus techniques available. The output signal of the camera 35 is connected to the processor 18.

When the camera is used to detect motion of the hand-held device, the camera controller 37 automatically selects a suitable resolution and refresh rate, so that the refresh rate is a flexible motion signal from changes in successive images. High enough to obtain. It is usually not necessary to use the complete image captured by the digital camera for motion detection. The software may select a particular section of the image for motion detection such that the amount of data to be processed is reduced. These two measures (low resolution and using sections of the image) allow for high sampling rates and reduced power consumption due to low data processing power requirements.

Some faces in the camera view may not be particularly suitable for detecting motion, for example because of uniform faces or because the distance from the camera view to objects is too far away to accurately determine the distance change. . These problems can be solved (in embodiments with adjustable camera orientation) by pointing the camera to another available object with sufficient texture, such as a user. The camera direction is thus reversed compared to the "normal" direction. Therefore, the motion signal obtained from the camera signal is automatically reversed when the camera is directed toward the user.

The camera 35 is a conventional digital camera so not all the features of the camera 35 are shown. The image sensor of the digital camera can be any of the known configurations for solid state image sensors, such as frame transmission, interline transmission CCDs or diode arrays.

Standard CCD devices are sensitive to both visible and near infrared light. Therefore, conventional digital cameras that capture images with visible light are provided with an infrared filter to prevent the influence of infrared light on the captured image. The mobile telephone 1 is also provided with an infrared filter (not shown). The infrared filter can be moved from the light path to the camera to capture infrared images. The IrDA port 38 may be set to emit continuously to serve as a light source for the camera 35 when the camera is in infrared mode. Thus, the camera 35 can be used even when there is little ambient light or no ambient light. Details of the device for moving the infrared filter into and out of the camera path as well as the use of an IrDA port as a light source for a digital camera and adjustments to an auto focus system are described in US patent application Ser. Is disclosed.

Preferably the lens 36 is a fixed focus lens with a group of lenses movable to allow auto focus, but the lens 36 adjusts to focus on different portions of the received image, as will be appreciated by those skilled in the art. It may be any type of lens provided.

The characteristics of visible light and near infrared light with respect to focusing are somewhat different. Therefore, the auto focus system has two settings; A first setting for capturing images with visible light and a second setting for infrared light.

Standard software for processing, storing and recalling images captured with visible light and infrared light is installed in the telephone 1. The software may itself be idiomatic and commercially available. The software can also control the refresh rate of images displayed on the display.

Optionally, the phone 1 can also include two tilt sensors 39 which determine the direction and magnitude of the displacement with respect to the vertical using the earth's gravity field. Such sensors may be of any of the well known types, such as those working with an encoding disk on a freely rotatable shaft connected to a weight, or of a type using an asymmetric additional provided sphere floating in liquid. Alternatively, the tilt sensor may be of a gyroscope type. The signals of the camera and the tilt sensors can be combined to generate a motion signal.

The telephone 1 according to a preferred embodiment adopts two modes of operation related to the use of the movement of the telephone to generate user input, referred to as navigation mode and fixed mode throughout this document. When the navigation mode is set, the movement of the telephone 1 is used as an input, and when displaying an image, the display view is automatically scrolled, zoomed and rotated according to the movement of the holding hand. The navigation mode is activated by pressing the left softkey 9 "Navigate". Thus, the navigation mode is activated and the function of the left softkey 9 is changed to "fixed". When reset to the fixed mode by pressing the left softkey "Fixed" again, the display view stops and no longer follows hand movements.

3 shows the relative azimuth axes in which the telephone 1 is rotated or translated to navigate an image on the display 3. Throughout this document, axis 40 will be referred to as the Y-axis, and the movement of the phone in the direction of the Y-axis is used to scroll images in the Y-axis in the preferred embodiment. Similarly the axis 41 will be referred to as the X-axis, and the movement of the phone in the direction of the X-axis is used in the preferred embodiment to scroll the images in the X-axis direction. The movement of the camera direction measured along the Z-axis 42 is used to control the zoom magnification of the images displayed on the display 3 in the preferred embodiment. Rotation about the Z-axis is used to rotate the images displayed on the display 3. Although these are the preferred functions assigned to the movements along the X-, Y- and Z-axes, any other function can be assigned to the movement in the direction of the axes or the rotational movement about the axes.

While scrolling, zooming, and rotating the display follows the movement of the device, the scroll / zoom / rotation ratio as well as the scroll / zoom / rotation ratio need not follow the exact amount of position change and can be reversed. As discussed below, the control software of the present invention can smooth the movement of the phone to provide a convenient effect on the display.

3.1 to 3.13 show an overview of the operation of the telephone 1 for scrolling, zooming and rotating images.

Referring to FIG. 3.1, a high resolution image of a holiday snapshot including a lake, bridge and mountaintop stored in ram 17a is shown for display on display 3. The display 3 is too small to show the entire image in a size large enough to conveniently view all the details in the image. The user presses the left softkey 9 " navigator " to enter the navigation mode. The phone activates the navigation mode and changes the label on the left softkey 9 to "fixed". In FIG. 3.2, the operator's hand is shown in the direction of the arrow 42 ′ in order to enlarge the view so as to show the center of the image with the leg in such a way that the display 3 is enlarged (FIG. 3.3). The navigation process is started when the phone 1 is moved along. By moving the phone along the Y-axis 40 in the direction of the arrow 40 ', the display 3 can scroll the images up so that the mountaintop on the bridge can be seen (Figure 3.5).

In FIG. 3.6 the holiday snapshot is indicated in the portrait orientation and can only be seen with the phone in the horizontal position. The user wants to view the image on the phone in an upright position such that the image is displayed in a horizontal position relative to the display 3. The user presses the left softkey 9 " navigator " to activate the navigation mode and rotates the phone half a turn counterclockwise (Fig. 3.7). The movement of the telephone 1 produces a series of altered images captured by the camera 35, from which the software on the telephone obtains a movement signal from it. Upon detection of a rotational movement about the Z-axis, the software on the phone rotates the displayed image in the opposite direction, and the software rotates the image half a clockwise direction when the user has finished rotating the counterclockwise direction. In this way, the user can conveniently view the image on the phone in an upright position. The image is thus static with respect to the fixed coordinate system in which the telephone 1 device is located.

The settings for the response to the movement of the phone 1 can thus be set in such a way that the user perceives the view as a view of the static image to which the magnification window (display 3) is moved.

3.8 shows the display 3 of the phone in the mode in which a graphical user interface is used to command the device. A window 70 containing a scrollable table is shown on the display 3. Scroll bars 71 and 72 are displayed on the right and bottom of the table, respectively. A cursor 73 can be moved on the display by moving the phone in the directions of the X-axis and the Y-axis. The left softkey 9 "left-click" has the same function as the left mouse button known from many window-based graphical user interfaces, in order to select the main function associated with the object represented by the cursor 73. By clicking, double-clicking or holding down the left softkey (9). The right softkey 9 "right-click" also has the same function as the right mouse button known from many window-based graphical user interfaces, in order to select auxiliary functions associated with the object represented by the cursor 73. . It is possible to assign the "left click" and "right click" to other keys of the phone, but using a similar layout, such as for the keys of a computer mouse, may facilitate user acceptance.

Example 1: The user wants to scroll the table to the right to see column E. By holding down the left softkey 9 "left-click" and moving the phone to the right along the X-axis, the table is scrolled by dragging to the right with the cursor as shown in Figure 3.9.

Example 2: The user wants to resize the window 70. The movement of the phone 1 in the direction of the Z-axis after positioning the cursor 73 on the upper bar of the window 70 and while holding the left softkey 9 "left-click" Adjust the size of the window 70. Moving the phone towards the user enlarges the window and moves away from the user as shown in FIG. 3.10 to reduce the size of the window (not shown).

Example 3: A user wants to move a portion of an image in an image editing program. The portion of the image to be moved is represented by a box 65 of considerable size. Moving the mouse by holding the left softkey 9 "left-click" down and rotating the phone about the X-axis and the Y-axis adjusts the size of the box (Figure 3.14). After adjusting the size of the box 65, the left softkey 9 is released and the cursor 73 can be freely moved. Position the cursor 71 on the box 65 and rotate the left softkey 9 "left-click" while rotating the phone about the Y-axis and / or X-axis until the box is moved to the desired position. By holding down the box can be dragged to the desired position. By releasing the left softkey 9 the box can be put down and the cursor 73 can move freely again. Thus, part of the image in the box is cut from the original position and pasted to the new position.

In the same way any object in a text editing program, for example texts or numbers / text in a spreadsheet, can be displayed with the left softkey, resized and dragged into "click and drag".

The most common "gestures" performed by moving the phone and pressing the softkeys are as follows:

Point (locating the cursor on an object of the graphical user interface),

Left-click (press and release the left softkey),

Double-click to activate the object on which the cursor is located (quickly holding down the softkey twice in succession),

Right-click (press and release the right softkey) to call up the context-sensitive menu,

Drag to move or resize the objects (holding the left softkey down while moving the phone to move the cursor).

3.11 to 3.13 illustrate another method for manipulating an enlarged portion of an image. 3.11 shows a display 3 with images of some advanced types of mobile phones. Magnification window 103 magnifies a portion of the image to allow the user a view with both a good overview and a possibility to see details in a selected portion of the image. To move the phone in the direction of arrow 41 'and / or arrow 40' (Fig. 3.12) to position the magnifying window 103 at the desired position (Fig. 3.13), the left softkey 9 "Navigate". By holding down the magnification window 103 can be moved on the display 3. The magnification of the magnification window can be changed by moving the phone 1 in the direction of the Z-axis (not shown).

Figure 3.16 illustrates another example of use of the present invention in the form of an application for entering text. At the top of the display a string of characters already entered is displayed. A set of characters that can be entered, in this example an alphabet, is displayed below the top of the display. Other character sets may include a set of numbers or a specific set of symbols.

The functions of the left softkey 9 "type" and the right softkey 9 "option" are displayed at the bottom of the display. One of the characters of the character set is indicated by bold print. By rotating the phone about the Y-axis 40 the indication moves to the left or to the right. By rotating the phone about the X-axis 41 the indication moves up and down. The displayed character is added to the string of characters by pressing the left softkey 9 "Type".

By pressing the right softkey “Options” a scrollable list of selectable menu items is displayed (not shown), the selectable menu items being “Delete last character”, “Screen clear”, “Numeric character set”, “Symbol” Character set " and " exit ", one of the menu items being indicated by reverse print. The list can be scrolled by rotating the phone about the Z-axis 41 and the displayed menu item is selected by pressing the left softkey 9 "Select".

FIG. 3.17 shows another example of the use of the present invention in the form of an application for inputting notes into staff 69. FIG. A cursor 73 (shape changed to cross for this application) is used to indicate the position where the note is to be input. By rotating the phone about the Y-axis 40 the cursor can be moved left or right. By rotating the phone about the X-axis 41 the cursor can move up and down.

The cursor is located above the position at which the note is to be entered, the higher tones are placed higher in the staff and the lower tones are lower in the staff. The pitch is input by pressing the left softkey 9 "Type". After typing a note, the application prompts for the length of the note by displaying the text "Length? 1 = 1 2 = ½ 4 = ¼ 8 = ⅛" between the labels for the staff and the softkeys. . The note length is input by pressing an alphanumeric key having a value associated with the desired note length.

By pressing the right softkey “Options” a scrollable list of selectable menu items is displayed (not shown), the selectable menu items being “Delete Last Note”, “Delete Staple”, “Enter Specific Note” and “Exit”. And one of the menu items is displayed in reverse print. The list can be scrolled by rotating the phone about the X-axis 41 and the displayed menu item is selected by pressing the left softkey 9 "Select".

Figure 3.18 illustrates another example of the use of the present invention in the form of an application that simulates a maze game. The player (a user of the hand-held device) uses the virtual ball (through the virtual maze formed by the virtual walls 57 on the virtual playing surface through the plurality of virtual openings 58 through which the virtual ball 59 can fall). 59) try to guide.

The application simulates the effect of tilting the actual playing surface off the horizontal plane caused by the gravitational force it has on the actual ball. For example, the virtual ball starts to roll down the display 3 when the actual playing surface is tilted out of the horizontal plane. In addition, the effect of gravity on the actual ball passing through the actual opening is simulated by the application. For example, the virtual ball falls through the virtual opening as it passes through the virtual opening.

At the start of the game, the display is maintained horizontally or in some other orientation that the user considers suitable as a reference orientation for calibrating the virtual "horizontal" position of the display. Movement of the virtual ball on the virtual playing surface is controlled by moving the telephone out of the horizontal plane and back into the horizontal plane by rotating the telephone about the Y-axis 40 and / or the X-axis. The rotational movement is detected from a change between successive images captured by the camera and translated into a change in the speed and rolling direction of the virtual ball.

Figure 3.19 shows another example of the use of the present invention in the form of another computer game, an application for controlling car racing simulation in this example. The player (user of the hand-held device) tries to "drive" the car around the race course as quickly as possible. The application allows the player to control coordination, braking and fueling functions. The virtual view on the race course is the main content of the display, but is not shown in FIG. The display further shows handles and rearview mirrors. The race is started by double clicking on the left softkey 9 "Run". The control settings for speed and direction are calibrated and set to zero for the current position and orientation of the phone 1. After the audible start signal the driver is assumed to attempt to follow the indicated race course. The "car" is adjusted by rotating the phone about the Z-axis 42. Rotating the phone clockwise in the calibrated position about the Z-axis causes the "car" to turn to the right. Rotating the phone further in the calibrated orientation makes the “car” turn more sharply and vice versa. Similarly, rotating the phone counterclockwise about the Z-axis in the calibrated position causes the "car" to turn left. The speed of the "car" is controlled by tilting the telephone 1 about the X-axis 41. Rotating the phone 1 in one direction in the calibrated position is used to provide "fuel". Further rotation of the phone 1 in the corrected orientation increases the amount of "fuel" and vice versa. The amount of braking supplied to the motor vehicle is similarly controlled by rotating the telephone 1 in the opposite direction in the calibrated position. The user can select which direction of rotation is set for the X-axis to provide "fuel", the braking direction always being reversed. Other computer games that can be controlled in a similar manner but use more motion / control axes include motorcycle racing and helicopter flight. For motorcycle racing, in order to create a very realistic experience, a factor balance can be included in relation to the movement on one of the axes, for example, the adjustment bar rotation is connected to the rotation about the Z-axis and provides fuel. And the braking is connected to the rotation about the X-axis and the balance is connected to the translational movement along the X-axis. In order to enhance the game, the ability of the mobile phone to produce sound through loudspeakers, in particular hands-free loudspeakers, is used for simulating motor and / or propeller sounds, for example. The vibration function (not shown) of the mobile phone can be used to provide feedback with respect to shocks and crashes.

Figure 3.20 illustrates another example of the use of the present invention in the form of an application for controlling the sound setting of the music player function of the telephone. The application allows the user to control the volume, bass and treble. The application displays a volume button, a bass button and a treble button on the display 3. The button to be operated is indicated by a thicker line (as indicated for the "volume" button in figure 3.20). The indication can be moved to other buttons by rotating the phone about the Y-axis 40. The indicated button is operated by holding down the left softkey 9 "control" while rotating the telephone about the Z-axis 42. The clockwise rotation of the phone results in an increased setting of the associated parameter and the counterclockwise rotation of the phone results in a reduced setting of the related parameter. The application is terminated by pressing the right softkey 9 "Exit". Alternatively, the display shows slide control knobs for each of the parameters to be controlled (not shown). The handle to be manipulated is indicated by a thicker line thickness. The indication can be moved to other handles by rotating the phone about the Y-axis 40. The indicated handle is operated by holding down the left softkey 9 "control" while moving the phone in the direction of the Z-axis 42. Moving the phone in the direction in which the display 3 is facing increases the associated parameter setting, and moving the phone in the opposite direction results in a reduced setting of the associated parameter.

Another example of use of the present invention relates to another terminal, such as a PC (not shown). The movement signal of the phone is transmitted to the PC to control the movement of the object. The object may be a three-dimensional object displayed on a screen connected to the PC, and the orientation changes of the phone are used to change the orientation of the displayed object. The orientation of the object on the screen can be completely synchronized with the orientation of the phone. After initial calibration of the relative position, for example, if the phone is standing upright, the displayed object also stands upright. For example, to display a product, the product can be displayed as an object on a large screen. To change the orientation of the object, the user changes the orientation of the telephone by rotating the telephone, and the PC rotates the displayed object in the same manner in response to the signal it receives from the telephone.

Another example of using the phone with another terminal, ie a workstation or a PC, is to move to a virtual three-dimensional space displayed on the screen, for example by a CAD program. The movement of the phone in a real three-dimensional world is included in the signal sent to the workstation and the PC or workstation converts the signal into a movement of the viewing position in the "three-dimensional space" displayed on the screen. The change in orientation of the phone is also included in the signal transmitted to the workstation and converted into a change in viewing direction in the "three-dimensional space" displayed on the screen. Thus, a user can "dial" through the virtual room by walking around in the real world while holding or carrying the phone, and change the viewing direction by pointing the phone in the desired viewing direction. In order to facilitate the use of this approach, the present invention can be included in a small device mounted on another type of headgear or included in a helmet, such that the viewing direction is changed by the user turning his / her head in the desired view direction. Can be.

The method of controlling the viewing position and the viewing direction may be applied to computer games and some other software applications displaying a virtual three-dimensional space on the screen for moving and observing through a room or some other place created in the virtual world. Can be used equally for Another example of the use of such a control method is a gaming application that guides, i.e., instructs the user to follow a virtual path and examines a motion signal when the user actually moves the path. Such games can be used by a group of users with interactive mobile phones in a suitable open space, such as a sports ground.

4 schematically shows a software flow diagram for manipulation of an image as described with reference to FIGS. 3.1 to 3.7. The flow from start 80 to end 98 is performed several times per second in the standard polling process of processor 18. At an initial stage in block 82, the current view settings are displayed. The label above the left softkey 9 is set to "navigation" in block 84. The state of the left softkey 9 is checked at block 86. If the left softkey 9 is pressed, the system is set to navigation mode at block 88 and the label above the left softkey is changed to "fixed".

In block 90 the camera image and auto focus settings are obtained, stored and compared with previous readings. If a change in image or auto focus setting is detected in block 92, the program obtains the movement of the phone 1 from the changes in block 94 and sets new view settings, i.e. zoom magnification, rotation angle and image to be displayed. Calculate the part of. It also instructs the display controller 31 to refresh the display 3 to display a new view, which stores the current camera image as a reference for comparison in a new iteration of the process.

At block 96 the status of the left softkey 9 is checked and if it is pressed the process ends at block 98 until the program is polled again. If the check for the left softkey 9 is negative, the program goes to step 90 and the process repeats itself.

The program may be set with different response curves to calculate a new view setting in response to the change in camera image and / or auto focus setting at block 94. Fine or coarse response modes can be set by the operator or dynamically changed while the system is in navigation mode. Through fine response, the display view changes at a relatively slow rate in response to the movement of the phone. With a coarse response, the display view changes quickly in response to the movement of the phone. The response can also be reversed to adapt to user preferences.

Functions associated with certain types of movement of the telephone 1 can be set. Thus, the user can set the program such that the cursor 73 can be moved up and down by rotation about the X-axis and left and right by rotation about the Z-axis, for example. have. Certain types of movement of the phone 1 may be associated with functions not related to the display 3, such as sound settings (e.g. volume and balance) and display settings (e.g. brightness, color balance and contrast). Can be.

5 and 6 show an alternative preferred embodiment of the hand-held device according to the invention in the form of communicator 101. The communicator 101 has a larger display 103 and a larger keyboard 107 but is basically formed in the same way as the mobile phone 1. The camera 135 (only the lens 136 is shown) is mounted on the back / bottom of the communicator and has the same function as the camera in the telephone 1. The internal hardware is also formed in the same way as the telephone 1 but with increased processing power and larger memory. The movement of the communicator 101 in the directions of the X-axis 141, the Y-axis 140, and the Z-axis 142 or the movement of the axes has the same function as in the telephone 1.

Although the present invention has been described in detail for purposes of illustration, it is understood that such details are for the purpose only and that changes may be made in the invention by those skilled in the art without departing from the scope of the invention.

Thus, although preferred embodiments of the apparatuses and methods have been described with reference to the environment in which they are developed, they are only intended to illustrate the principles of the invention. Other embodiments and configurations may be devised without departing from the scope of the appended claims.

Claims (48)

In a hand-held device, A processor, a digital camera for capturing moving images or still images, and means for converting signals from the camera into motion signals representing movement of the hand-held device. The hand-held device of claim 1, further comprising a user interface wherein the movement of the hand-held device is used as a user input via an acquired movement signal of the hand-held device. A hand-held device according to claim 1 or 2, further comprising a display suitable for displaying the captured images. 4. The method of claim 3, wherein the movement of the hand-held device at least partially displays images on the display by moving the images in a manner substantially corresponding to the movement of the hand-held device. Hand-held device, characterized in that used to. 5. The method of claim 4, wherein the type of movement of the hand-held device is used to move, and / or zoom, and / or zoom and / or rotate images displayed on the display. Hand-held device. The method of claim 5, wherein the movement substantially parallel to the plane of the display of the hand-held device is used to scroll the image displayed on the display and / or the movement substantially perpendicular to the plane of the display. Hand-held device, wherein the hand-held device is used to zoom an image displayed on and / or the rotational movement of the hand-held device. The hand-held device according to claim 4, wherein the images are images previously captured by the camera. The hand-held device according to claim 4, wherein the movement of the image is reversed with respect to the movement of the hand-held device. The hand-held of claim 2, wherein the user interface comprises a graphical user interface, wherein movement of the hand-held device is used as input to the graphical user interface. Device. 10. The method of claim 9, wherein by moving the object in a manner substantially corresponding to the movement or inverted movement of the hand-held device, movement of the hand-held device causes the object to be displayed by the graphical user interface. Hand-held device, characterized in that the object used for the manipulation and displayed by the graphical user interface may be an icon, dialog box, window, menu or pointer. 10. The method of claim 9, wherein the type of movement of the hand-held device is used to move, and / or zoom, and / or zoom and / or rotate the objects represented by the graphical user interface. Hand-held device. The method of claim 11, wherein the motion substantially parallel to the plane of the display of the hand-held device is used to scroll the object represented by the graphical user interface, and / or the motion substantially perpendicular to the plane of the display is Hand-held device, which is used to zoom an object displayed by a graphical user interface and / or a rotational movement of the hand-held device is used to rotate an object displayed by the graphical user interface. 13. A hand-held device according to any one of claims 2 to 12, wherein said digital camera is detachable. 14. A hand-held device according to any of claims 2 to 13, wherein the digital camera is movable relative to the hand-held device. The method of claim 2, wherein the means for converting the signal from the camera into a motion signal comprises moving the continuous image captured by the camera or changes between portions of the continuous images. Hand-held device, characterized in that acquiring a signal. 16. The hand-held device according to any one of claims 2 to 15, wherein the camera has an auto focus system, and the focusing setting of the auto focus system is used to detect movement in the camera direction. 17. A hand-held device according to any one of claims 2 to 16, further comprising at least one key, wherein the function of the movement type depends on the state of the at least one key. 18. The rotational movement of the hand-held device with respect to an axis substantially perpendicular to the display is preferably in relation to the fixed coordinate system in which the hand-held device is located. Hand-held device in such a way that an image or object becomes static, resulting in a reverse rotational movement of the image or graphical user interface object with respect to the display. 19. A hand-held device according to any one of claims 2 to 18, wherein said motion signal is used to adjust device settings, said device settings preferably comprising sound settings and display settings. 20. The apparatus of any one of claims 9 to 19, further comprising a keypad having at least a first key and a second key, wherein the graphical user interface includes a cursor, wherein movement of the hand-held device causes the cursor to move. Hand-held devices, wherein the primary functions associated with the associated object and used for positioning on an object of the graphical user interface are activated by pressing the first key and the auxiliary functions associated with the associated object are activated by pressing the second key. . 21. The method of claim 20, wherein the function of the first key is related to the selection and activation of objects of the graphical user interface, preferably the function of the second key is related to invoking a context-sensitive menu. Hand-held device. 22. The hand according to claim 21, wherein the selection of the related object is performed by pressing and releasing the first key, and the activation of the related object is preferably performed by pressing and releasing the first key twice in quick succession. -Held device. 23. A hand-held according to claim 21 or 22, wherein the movement or resizing of the associated object is performed by holding the first key down while moving the hand-held device to move the cursor. Device. 24. The method according to any one of claims 20 to 23, wherein the first and second keys are softkeys and the current function of the softkeys is shown in dedicated fields of the display, preferably of the display. Hand-held device characterized in that. 25. The device of claim 24, wherein the first key is disposed below the display, on the left side of the display, preferably near the bottom edge of the display, and the second key is preferably near the bottom edge of the display. A right-hand side of the display, below the display. 26. The system of any one of the preceding claims, further comprising at least one gravity based tilt sensor, wherein the signal from the at least one tilt sensor is in concert with a signal from the camera to generate the motion signal. Hand-held device, characterized in that used as. 27. The hand-held device of claim 26, wherein the tilt sensor is associated with the X-axis and / or the tilt sensor is associated with the Z-axis. 28. The hand-held device of claim 27, wherein the signal from the at least one tilt sensor is used to determine the absolute orientation of the hand-held device with respect to the direction of gravity attraction. 29. A hand-held device according to any preceding claim, further comprising means for transmitting the movement signal to another terminal via a cable, an infrared wave or a radio frequency wave. 30. A system comprising a hand-held device according to claim 29 and a terminal capable of displaying virtual three-dimensional objects on a two-dimensional screen, the terminal in response to signals received from the hand-held device. Means for changing the orientation of the hand-held device, wherein the orientation change of the hand-held device is translated into a corresponding orientation change of the displayed object. 31. The system of claim 30, wherein the change of position of the hand-held device translates into a change of position of the displayed object. 30. A system comprising a hand-held device according to claim 29 and a terminal capable of displaying a virtual three-dimensional space on a two-dimensional screen, said terminal in response to signals received from said hand-held device. Means for changing the viewing position in dimensional space, wherein the position change of the hand-held device is translated into a corresponding change in the viewing position. 31. The system of claim 30, wherein a change in orientation of the hand-held device is translated into a corresponding change in the viewing direction in the virtual three-dimensional space. A method for generating user input to a hand-held device having a processor, a user interface and a digital camera for capturing video or still images, the method comprising: Determining movement of the hand-held device from the camera signal; And Using the determined movement of the hand-held device as input to the user interface. Use of a digital camera to capture video or still images of the hand-held device with a processor to generate a motion signal indicative of the movement of the hand-held device. In a hand-held device, A processor, means for detecting movement of the hand-held device, a display, a keypad with at least a first key and a second key, a graphical user interface with objects and a cursor, and a sensed movement of the hand-held device. Means for converting a cursor into a signal suitable for moving on the display. 37. The method of claim 36, wherein movement of the hand-held device is used to position the cursor on an object of the graphical user interface and main functions associated with the related object are activated by pressing the first key and associated with the related object. Secondary functions are activated by pressing the second key. 38. The method of claim 37, wherein the function of the first key is related to the selection and activation of objects of the graphical user interface, and the function of the second key is preferably associated with invoking a context-sensitive menu. Hand-held device. The method of claim 38, wherein the selection of the related object is performed by pressing and releasing the first key, and activation of the related object is preferably performed by pressing and releasing the first key twice in quick succession. Hand-held device. 40. The method of claim 38 or 39, wherein moving or resizing the associated object is accomplished by holding the first key down while moving the hand-held device to move the cursor and the associated object in line with the cursor. Hand-held device, characterized in that performed. 41. The method of any of claims 36-40, wherein the first and second keys are softkeys and the current function of the softkeys is shown in dedicated fields of the display, preferably the display. Hand-held device characterized in that. 42. The device of claim 41, wherein the first key is disposed below the display, on the left side of the display, preferably near the bottom edge of the display, and the second key is preferably near the bottom edge of the display. A right-hand side of the display, below the display. 43. The device according to any one of claims 36 to 42, wherein the means for converting movement of the hand-held device into a signal suitable for moving the cursor on the display comprises a tilt sensor and / or an image capture device and / or A hand-held device comprising an accelerometer. 44. The hand-held device of claim 43, wherein the image capture device is a video or still image digital camera. 45. The hand-held of any one of claims 36 to 44, further comprising means for converting signals from the camera and / or tilt sensor and / or accelerometer into position signals for the cursor. Device. 46. The method of claim 45, wherein the means for converting the signal from the camera into a motion signal obtains the motion signal from changes between successive images or portions of successive images captured by the camera. Hand-held device. 45. The hand-held device of claim 44, wherein the camera has an auto focus system, and the focusing setting of the auto focus system is used to detect movement in the camera direction. 48. A hand-held device according to any of claims 36 to 47, wherein the graphical user interface comprises one or more object types of icons, dialog boxes, windows, menus and pointers.