KR101066954B1 - A system and method for inputting user command using a pointing device - Google Patents

Abstract

In a user command input system including a pointing device and an image device for moving a pointer on a display screen and selecting an object according to an intention of a user according to an embodiment of the present invention, the image device may be arranged on the display screen. Displaying at least one first object belonging to one layer, and receiving a command for selecting one of the first objects from the pointing device, at least one second belonging to a second layer around the selected object Display the object.

Pointing Devices, Motion Sensors, Gyro, Accelerometers, Objects, Selection Commands, Layers

Description

A system and method for inputting user command using a pointing device}

The present invention relates to a user command input system using a pointing device having a motion sensor. More particularly, the present invention relates to a user command input system using a pointing device used to input a command in a video display device such as a digital TV or a set-top box. A method for easily selecting objects and executing corresponding commands.

As the field of computer science has developed, a variety of devices have been developed that allow users to enter information into computer devices. One group of such devices is called a pointing device. As the user manipulates the components of this device, position data corresponding to the movement of the pointing device is generated. This position data is then converted into movement of the pointer image appearing on the display. Thus, by moving the pointing device, the user can associate the pointer image with the objects displayed on the display. The object refers to a user interface that performs a specific operation when a menu, a button, an image, or the like is selected. Thereafter, the user may perform a specific command related to the object through a selection operation such as pressing a specific button of the pointing device.

Users of general personal computers use Microsoft's Windows as an operating system (OS) to run their own computers. This is due to convenient mouse functions and various graphic functions that are not supported by the existing DOS (Disk Operating System) system. When a user enters a specific command through a window, the user can drag, scroll, or click the mouse without typing a keyboard. You can easily enter a command by, for example.

On the other hand, in a video display device in which a keyboard or a mouse used in a personal computer such as a digital TV or a set-top box cannot be used, various commands are input using a keypad provided on a remote control device called a remote controller. have. The reason why the keypad press input method is mainly used is that, unlike a personal computer, such a device is not fixed to a specific location for the user to operate the device, and because the device requires manipulation in an open space such as a living room, This is because it is difficult to use an input means mainly fixed to a plane such as a mouse.

For this reason, the input means in the image display device as described above is preferred to the input means easy to operate in a space such as a remote control having a keypad. However, since the keypad input method uses a method of moving only one space when the keypad is pressed in character input or menu selection, there is a limitation in the user's command input.

In view of these problems, recently, three-dimensional pointing devices equipped with motion sensors such as gyroscopes (hereinafter, referred to as gyroscopes) and accelerometers have been developed. By moving the three-dimensional pointing device, the user can move the pointer image on the display corresponding to the desired direction and speed, and select and execute a desired object by pressing a specific button provided on the pointing device.

 However, even when a three-dimensional pointing device that is not limited in space is provided, there is a problem that command input is difficult to be made quickly and easily as compared to a keyboard or a mouse of a personal computer. For example, even if a virtual keyboard is provided on the display, the distance between input phonemes is not always close, so character input is not easy, and even if a particular item is selected from menus and submenus, the moving distance is not minimized. Execution of certain commands is not easy.

1 and 2 show that selecting an object using a conventional pointing device is inefficient. FIG. 1 is a diagram showing an example of a conventional technique of selecting a phoneme from a dial 10 displayed on a display to input characters. Here, if the user wants to input the letter "large", the user first selects the consonant "c" (11), moves the three-dimensional pointing device to the right, selects the vowel "ㅏ" (12), and then moves the device downward to the left. Move to select the vowel “ㅣ” (13). As such, since the moving distance from selecting one phoneme to selecting another phoneme is arbitrarily changed, considerable effort and effort is required for the user to complete the character input.

2 is a diagram illustrating an example of the prior art of selecting a final item while moving from a menu on a display to a lower menu. The user selects the category "movie" 21 in the main menu 22, again selects the genre "action" 23 in the first sub-menu 24, and selects "" in the second sub-menu 26. Assuming a movie called "Matrix" 25 is selected, a pointer movement corresponding to a total of eight intervals is required. Since the pointer may be insufficient in terms of user operability rather than a mouse, the user may feel inconvenient if the variable distance has to be moved in selecting a specific object on the display using the pointing device.

The present invention seeks to provide a more convenient method for selecting an object on a display using the pointing device, in view of the operational specificity of the pointing device.

The technical problem to be achieved by the present invention is to minimize the burden on the user of the pointing device having a motion sensor.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above technical problem, according to an embodiment of the present invention, a pointing device for detecting a user's movement through a motion sensor, and moving the pointer on the display screen according to the detected user's movement to select an object In a user command input system including an imaging device, the imaging device displays at least one first object belonging to a first layer on the display screen, and selects one of the first objects from the pointing device. Upon receiving the command, at least one second object belonging to a second layer is displayed around the selected object.

In order to achieve the above technical problem, according to an embodiment of the present invention, a user command input method for detecting a user's movement by a pointing device having a motion sensor and controlling a pointer on a display screen according to the detected movement, Detecting a movement of a user, moving a pointer on a display screen according to the detected movement, selecting an object, displaying at least one first object belonging to a first layer on the display screen, and Receiving a command for selecting one of the first objects from a pointing device and displaying at least one second object belonging to a second layer around the selected object.

According to the user command input system and method using the pointing device according to the present invention, the operation burden when a user using the pointing device having a motion sensor moves between each layer belonging to the objects displayed on the display screen Can be minimized.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

3 and 4 illustrate a concept of selecting an object on a display using a pointing device having an inertial sensor according to an exemplary embodiment of the present invention. In the present invention, an object refers to a user interface selected on a display for performing a specific operation such as a menu, a button, an image, or the like.

As shown in FIG. 3, when the object 30 belonging to the main menu is selected by the pointing device, the objects 31 to 36 of the first submenu are displayed and activated in the vicinity of the object 30. . At this time, the user can select from among the objects 31 to 36 currently active. In FIG. 3, the sub-objects 31 to 36 are circularly arranged around the object 30, but only one example may be arranged in other forms as long as they are arranged to be close to the object 30. It will be understood by those skilled in the art.

In addition, as shown in FIG. 4, when the user issues a selection command for one object 32 among the activated objects 31 to 36, the objects 41 to 45 of the second submenu are displayed. Displayed and activated in the vicinity of 32, the user can likewise select from the activated objects. If the user decides not to proceed to further submenus (or if there are no further submenus), the user may issue a selection command for the final object to define a predefined command corresponding to the final object ( Or the operation) is performed. In the present invention, the selection command may be triggered by pressing a specific button of the pointing device or by a predefined specific gesture. For example, the specific gesture may include an operation of stabbing forward in a position in which the pointing device is held. This stab motion can be easily detected by, for example, an accelerometer aligned in the longitudinal direction of the pointing device, which makes it easier for the user to take action and to determine if it is mapped to a defined gesture compared to other complex gestures. There is this. However, in order to prevent a malfunction, it is preferable to determine the gesture for the selection command only when the distance, speed, or acceleration exceeds a certain threshold. In addition, a selection command may be transmitted by simply placing the pointer on a specific object for a predetermined time without additional manipulation. If a pointer is placed on a specific object for a predetermined time or more, the submenu is recognized as an intermediate selection. When a specific button is pressed while being displayed, the selection command may be defined as completed.

As described above with reference to FIGS. 3 and 4, since objects belonging to each level among a plurality of menu levels (main menu, first sub-menu, second sub-menu, etc.) are activated / deactivated at the same time, an image related to the objects. And the attributes are preferably separated into layers. Therefore, if it is necessary to activate objects for a particular menu level, it can be easily implemented by activating a layer corresponding to the menu level.

As described above, minimizing the distance to move from one object to the next object is not merely for reducing the user's movement. Since the pointing device operated by the motion sensor is free to move in three-dimensional space and there is no reference plane, it is relatively difficult to control the movement of the pointer as compared to a conventional input device such as a two-dimensional mouse or a keypad. It is. In the conventional input devices, even if one object is selected and then a long distance is moved to select another object, there is no great difficulty in controlling. Of course, the accuracy of the pointer's control will vary depending on the motion sensor's own accuracy of detecting acceleration and deceleration while moving the pointer, and algorithms calculated from the output of the motion sensor, but the precision of moving the pointer farther than the two-dimensional mouse It is well known that it is relatively inferior to the back. Therefore, in the pointing device using the motion sensor as in the present invention, it is very important to minimize the distance between the selected objects.

5 is a schematic diagram illustrating a user command input system 300 according to an embodiment of the present invention. The user command input system 300 includes at least one pointing device 100 and at least one imaging device 200. The pointing device 100 detects a user's movement through a motion sensor and converts the detected result into a horizontal movement and a longitudinal movement on a display plane, and transmits the image to the imaging apparatus 200 according to a predetermined wireless communication protocol. It plays a role. The wireless communication protocol may include Bluetooth, IEEE 802.11 WLAN standard, IEEE 802.15.3 wireless fan standard, and other wireless communication protocols.

The imaging apparatus 200 receives the detected result data and moves the pointer on the display based on the received data. In addition, when a selection command is transmitted from the pointing device 100, a command corresponding to the object pointed to by the pointer is executed. The imaging apparatus 200 may include any imaging apparatus that communicates with the pointing device 100 remotely and directly or indirectly with a display device, regardless of its name, such as a digital TV, a set-top box, an IPTV, or the like.

FIG. 6 is a block diagram illustrating in detail the configuration of the pointing device 100 and the imaging device 200 of FIG. 5. The pointing device 100 may be configured to include a motion sensor unit 110, a controller 120, a trigger detector 130, a coordinate calculator 140, and a transmitter 150.

The motion sensor unit 110 detects the direction of movement of the user (pointing device 100) and provides a sampled output value (digital value). In detail, the motion sensor unit 110 may include at least one gyroscope or at least one accelerometer or a combination of a gyroscope and an accelerometer. The gyroscope may be used regardless of mechanical, fluid, optical, or piezoelectric type, and the accelerometer may be used regardless of a piezodlectric type or a moving coil type.

The coordinate calculator 140 calculates a horizontal movement value (x displacement) and a longitudinal movement value (y displacement) on the display screen from the output value provided from the motion sensor unit 110.

For example, when the motion sensor unit 110 uses a gyroscope, the motion sensor unit 110 generates angular velocity information W _bx , W _by , and W _bz corresponding to a user's movement, and coordinate coordinates ( 140). The subscript b means acceleration information in the body frame, and the subscripts x, y, and z mean information about the x-axis, the y-axis, and the z-axis. The coordinate calculator 140 converts the input angular velocity information into rotation angle information through a predetermined calculation process. The rotation angle information includes at least one of pitch φ, roll θ, and yaw ψ. In the following reference frame composed of x, y, and z as shown in FIG. 7, the pitch φ is rotated in the y-axis direction, the roll θ is rotated in the x-axis direction, and the yaw is the z-axis. Each means rotation about the direction. The criteria for determining positive and negative values for these rotations follow the right-handed coordinate system. Since a calculation process for converting the rotation angle information belongs to a known technique, a detailed description thereof will be omitted herein. For example, the yaw may correspond to the lateral displacement on the display screen, and the pitch may correspond to the longitudinal displacement.

On the other hand, when the motion sensor unit 110 uses an accelerometer, the motion sensor unit 110 generates acceleration information (A _bx , A _by , A _bz ) corresponding to the user's movement and transmits it to the coordinate calculator 140. to provide. The coordinate calculator 140 double-integrates the acceleration information and converts the displacement information into respective displacement information about the x, y, and z axes. If a gyroscope and an accelerometer are used together, it is possible to calibrate the other based on the result of one of these calculations.

Regardless of the posture of the pointing device 100, the movement on the body frame may be matched to the movement on the display, and the movement on the navigation frame fixed in space may be matched to the movement on the display. .

The coordinate calculator 140 calculates displacement information (eg, horizontal displacement and longitudinal displacement) on the display and provides the same to the controller 120 through the above process. As such, providing the coordinate operation unit 140 in the pointing device 100 is one example, and the pointing device 100 simply provides the imaging device 200 with raw data relating to a sensor output relating to movement in space. In addition, an embodiment in which a coordinate calculation unit provided in the imaging device 200 calculates a lateral displacement and a longitudinal displacement from the raw data may be considered.

The trigger detector 130 detects a selection command from the user and transmits the selection command to the controller 120. The selection command may be detected by simply pressing a specific button provided in the pointing device 100, but may also be detected by determining whether a predetermined specific gesture has occurred. The specific gesture may include an operation of stabbing forward in a posture of holding a pointing device. Such a gesture has an advantage of allowing a user to intuitively press a button. Such a stabbing operation can be simply detected by, for example, an accelerometer (built in the motion sensor unit 110) aligned in the longitudinal direction of the pointing device 100.

The controller 120 controls the overall operation of the other components of the pointing device 100, and provides displacement information (lateral displacement and longitudinal displacement) provided from the coordinate calculating unit 140 and provided from the trigger detecting unit 130. The selection command is provided to the transmitter 150.

The transmitter 150 converts the displacement information and the selection command provided from the controller 120 into a wireless signal that conforms to a predetermined wireless communication protocol and transmits the converted signal to the video device 200.

The image device 200 may include a receiver 210, a controller 220, a memory 240, and an image processor 250.

The receiver 210 receives a radio signal transmitted from the pointing device 100 and restores displacement information and a selection command from the received radio signal.

The controller 220 controls the overall operation of the other components of the imaging apparatus 200 and controls the movement of the pointer on the display (not shown) of the imaging apparatus 200 by appropriately scaling the displacement information. According to the selection command, objects belonging to a lower layer of the object (current object) on which the current pointer is located are displayed around the current object (via the image processor 250), or an image device corresponding to the object where the current point is located. Operation 200 (video playback, text input, etc.) is performed.

As shown in FIGS. 3 and 4, the memory 240 stores a plurality of objects for each layer, and provides the objects for each layer at the request of the image processor 250.

The image processor 250 displays objects belonging to one layer on a display (not shown), and when a specific object is selected by the control of the controller 220, objects belonging to a lower layer of the object may be surrounded by the specific object. In addition to. In addition, the image processor 250 moves the pointer on a display (not shown) under the control of the controller 220.

Until now, each block in FIG. 6 is a software, such as a task, a class, a subroutine, a process, an object, an execution thread, a program, or a field-programmable gate array (ASGA) or an application (ASIC) that is performed in a predetermined area of memory. It may be implemented in hardware, such as a -specific integrated circuit, or may be a combination of the above software and hardware. In addition, each block may represent a portion of a module, segment, or code that includes one or more executable instructions for executing a specified logical function (s). It is also possible in some alternatives that the above-mentioned functions occur out of order. For example, the two blocks shown in succession may in fact be executed substantially concurrently, and the blocks may sometimes be executed in the reverse order, depending on the corresponding function.

8 to 13 illustrate embodiments in which the image processor 250 displays an object for each layer and allows a user to select a particular object among the layers according to an embodiment of the present invention.

8 to 10 illustrate an operation related to a character input application according to one embodiment of the present invention.

In order to select one of the objects (phonemes) included in the keypad 50 displayed on the display of the imaging device 200, the user uses the pointing device 100 on the pointer 52 on the desired object 52. Move) to the Select command. Then, objects (sub-objects) belonging to the lower layer of the selected object 52 are displayed around the selected object 52. In addition, a list 55 of various candidate items starting with the phoneme “” 57 corresponding to the selected object may be displayed on one side.

The sub-objects 64 displayed around the user after selecting the specific object 52 are displayed as shown in FIG. 9. As such, when the lower objects 64 are displayed, it is preferable that the upper objects are not displayed on the display screen or displayed in an inactive state so that they cannot be selected.

Since the particular object 52 is a consonant, the sub-objects 64 are a collection of vowels. However, in consideration of the limitation of the number, only short vowels are displayed in the vowels. When a user issues a selection command by moving the pointer 54 on one of the collections "ㅏ" 62, as shown in FIG. 10, sub-objects for the selected object 62 are surrounded by the selected object 62. Is displayed. Along with this, a list 65 of candidate items beginning with "C" 67, which is a combination of phonemes selected so far, is displayed on one side.

Sub-objects 74 for the vowel " k " 62 are twin consonants based on the vowel 62. The user may select "ㅐ" 72 of these dipoles to finally list only those items that begin with "large" 77 in the item list 77 on the right, where the desired item is finally selected. Can be.

11 to 13 illustrate an operation related to a content execution application according to another embodiment of the present invention.

The user may select one of the categories of content in the main menu (main layer) by moving the pointer 54 using the pointing device 100. When the user selects the object 81 called "movie", sub-objects are displayed around the object 81 as shown in FIG. Of course, at this time, the objects of the main menu can be displayed so that they are not selected or displayed as inactive.

The sub-objects are, for example, genres of the object 81 called "movie". When the user selects an object 82 called "action" among these genres, sub-objects representing the title of the movie are displayed around the object 82 as shown in FIG. The user can select and watch one of the "matrix" 84.

14 is a flowchart illustrating a user command input method using a pointing device according to an embodiment of the present invention. The method is a user command input method of detecting a user's movement using a pointing device and controlling a pointer on a display screen according to the detected movement.

First, the pointing device 100 detects a user's movement (S1). Then, the detected motion is transmitted to the video device. In the step S1, for example, the motion sensor unit 110 detects a user's movement to provide a sampled output value, and the coordinate calculator 140 calculates displacement information on the display screen from the output value. And detecting, by the trigger detecting unit 130, a command for selecting a specific object from a user, and transmitting, by the transmitter 150, the calculated displacement information and the selecting command to the video device 200. It may consist of steps.

Next, the controller 220 of the imaging apparatus 200 moves the pointer on the display screen according to the detected movement and a user's selection command (S2). In a preferred embodiment of the present invention, the selecting command is a gesture stabbing in the longitudinal direction of the pointing device.

Then, the image processor 250 displays at least one first object belonging to the first layer on the display screen (S3).

Thereafter, the receiver 210 receives a command for selecting one of the first objects from the pointing device 100 (S4). According to the first embodiment of the present invention, the at least one first object may be a consonant, and the at least one second object may be a vowel that forms a letter by combining with the consonant. Of course, even when the first object is a consonant, the second object does not necessarily need to be a vowel, and may be a consonant which can designate necessary information using the consonant of the first object. Further, according to the second embodiment of the present invention, the at least one first object is an upper classification of predetermined content, and the at least one second object is a lower classification of the content.

If the next layer of the first layer does not exist (NO in S5), the controller 220 executes an operation corresponding to the selected object in the first layer (S6).

Meanwhile, if the next layer of the first layer exists (YES in S5), the image processor 250 displays at least one second object belonging to the second layer around the selected object (S7). The second object may be displayed in a circle around the selected object. Thereafter, the S4 and subsequent steps may be repeated again.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

1 is a view showing an example of a conventional technique of selecting a phoneme from a dial 10 displayed on a display to input characters.

2 shows an example of the prior art of selecting the final item while moving from a menu on the display to a submenu.

3 and 4 illustrate the concept of selecting an object on a display using a pointing device with an inertial sensor in accordance with one embodiment of the present invention.

5 is a schematic diagram illustrating a user command input system according to an embodiment of the present invention.

6 is a block diagram illustrating in more detail the configuration of the pointing device 100 and the imaging device 200 of FIG. 5.

7 is a diagram explaining the definition of pitch, roll, and yaw;

8 to 10 are diagrams illustrating operations relating to a character input application according to one embodiment of the present invention.

11 to 13 illustrate operations related to a content execution application according to another embodiment of the present invention.

14 is a flowchart illustrating a user command input method using a pointing device according to an embodiment of the present invention.

(Symbol description of main part of drawing)

100: pointing device 110: motion sensor unit

120, 220: control unit 130: trigger detection unit

140: coordinate calculator 150: transmitter

200: video device 210: receiver

240: memory 250: image processor

Claims (19)

A user command input system comprising a pointing device for detecting a movement of a user and an image device for moving a pointer on a display screen and selecting an object according to the detected movement of the user. Displaying at least one first object belonging to a first layer on the display screen, and receiving a command to select one of the first objects from the pointing device, the second layer around the selected object Display at least one second object to which it belongs, And when at least one second object belonging to the second layer is displayed, the at least one first object belonging to the first layer is not displayed or inactive. The method of claim 1, And the at least one first object is a consonant, and the at least one second object is a vowel that combines with the consonant to form a character. The method of claim 1, And the at least one first object is a consonant, and the at least one second object is a consonant capable of specifying necessary information together with the consonant. The method of claim 1, And the at least one first object is an upper classification of predetermined content, and the at least one second object is a lower classification of the content. The method of claim 1, wherein the imaging device And when an object of the at least one second object is selected, executing an operation corresponding to the selected object among the at least one second object. The method of claim 1, wherein the second object A user command input system using a pointing device, which is displayed in a circle around the selected object. The method of claim 1, The pointing device is A motion sensor unit which detects a user's movement and provides a sampled output value; A coordinate calculator configured to calculate displacement information on the display screen from the output value; A trigger detector for detecting a command for selecting a specific object from a user; And And a transmitter for transmitting the calculated displacement information and the selected command to the video apparatus. The method of claim 7, wherein The imaging device A receiver which receives the displacement information and the command to select from the transmitter; A control unit controlling a movement of the pointer on the display screen based on the displacement information; A memory for storing images relating to the first and second objects; And And an image processor configured to display an image related to the first object or the second object on the display screen under the control of the controller. delete A user command input method of detecting a user's movement by a pointing device having a motion sensor and controlling a pointer on a display screen according to the detected movement, Detecting movement of a user; Moving a pointer on a display screen and selecting an object according to the detected movement; Displaying at least one first object belonging to a first layer on the display screen; Receiving a command to select one of the first objects from the pointing device; And Displaying at least one second object belonging to a second layer around the selected object, And when at least one second object belonging to the second layer is displayed, the at least one first object belonging to the first layer is not displayed or inactivated. The method of claim 10, The at least one first object is a consonant, and the at least one second object is a vowel that forms a character by combining with the consonant. The method of claim 10, And the at least one first object is a consonant, and the at least one second object is a consonant capable of designating necessary information using the consonant. The method of claim 10, And the at least one first object is an upper classification of a predetermined content, and the at least one second object is a lower classification of the content. The method of claim 10, And if an object of the at least one second object is selected, executing an operation corresponding to the selected object among the at least one second object. The method of claim 10, wherein the second object A user command input method using a pointing device, which is displayed in a circle around the selected object. The method of claim 10, Detecting the movement of the user Detecting a user's movement and providing a sampled output value; Calculating displacement information on the display screen from the output value; And And detecting a command for selecting a specific object from a user. delete A user command input system comprising a video device for moving a pointer on a display screen and selecting an object, wherein the video device includes: Displaying at least one first object belonging to a first layer on the display screen, and receiving a command to select one of the first objects from a pointing device, belonging to a second layer around the selected object Display at least one second object, The selecting command is a gesture stabbing in the longitudinal direction of the pointing device, And when at least one second object belonging to the second layer is displayed, the at least one first object belonging to the first layer is not displayed or inactive. A user command input method of controlling a pointer on a display screen according to a movement of a pointing device, Detecting movement of a user; Moving a pointer on a display screen and selecting an object according to the detected movement; Displaying at least one first object belonging to a first layer on the display screen; Receiving a command to select one of the first objects from the pointing device; And Displaying at least one second object belonging to a second layer around the selected object, The selecting command is a gesture stabbing in the longitudinal direction of the pointing device, And when at least one second object belonging to the second layer is displayed, the at least one first object belonging to the first layer is not displayed or inactivated.

Images