JP2010067143A - Operation and control device for graphical user interface - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation and control device for a graphical user interface. <P>SOLUTION: The operation and control device includes an image sensor unit and the graphical user interface. An image sensor has an infrared light emission device, an image intake device, and an arithmetic control module, and the infrared light emission device emits infrared light toward a user. The image intake device makes light rays including the infrared light reflected from the user pass, optically forms an image, and outputs a digital image signal. The arithmetic control module receives the digital image signal, performs arithmetic processing, and generates an operation/control signal according to diverse body movement of the user. The graphical user interface is displayed on a screen, and is connected to the image sensor unit. By the operation/control signal generated from the image sensor unit, the user can operate/control the graphical user interface by use of one's own body movement. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a graphical user interface (GUI) operation / control device, and in particular, by sensing various movements of the user's body, the user can operate / control the graphical user interface according to the movement of his / her body. It is related with the operation control apparatus to make.

  First, a graphical user interface is a graphics-based user interface that uses certain graphics and operating methods, for example, combining windows, menus, and cursors, between a user and a computer system. The basis for interaction. In this way, a user who is poor at command input can also input to the computer system. The user can contribute to understanding and searching for a general function, and can make the user know the operation of another program or function quickly by a similar method.

  Since the late 1980s, the application of graphical user interfaces has already matured. It is found in various electronic products such as desktop computers, notebook computers, mobile communication devices, personal digital assistants, satellite navigation system devices, etc., providing users with convenient, easy-to-use and quick operation methods. ing. However, when operating a computer system using a graphical user interface or interacting with a computer system, the user must still input in cooperation with a keyboard, mouse, touch panel or other operating / control device. In addition to limiting the development of graphical user interfaces, it is also not possible to provide a scenario-type operating environment.

  In a graphical user interface that senses and operates / controls existing user's physical movements, the user still needs to wear or use a specific input device, such as a controller or a remote control. Therefore, the user's specific movement cannot be accurately reacted, and the specific cursor on the display screen of the computer system or the computer game may not be sensitive to the user's physical movement or may be delayed. is there.

  As a result, the present inventor has intensively studied in view of the above-mentioned drawbacks, and as a result of blending the operation of science, the present invention has been completed, which has a rational design and can effectively improve the above problems.

  An object of the present invention is to provide a graphical user interface (GUI) operation / control device, and to operate a corresponding graphical user interface by sensing a user's physical movement. .

  In order to achieve the above object, a graphical user interface operation / control apparatus according to the present invention includes an image sensor unit and a graphical user interface. The image sensor includes an infrared light emitting device, an image capturing device, and an arithmetic control module. The infrared light emitting device is for emitting infrared rays toward a user. The image capturing device receives an infrared image by optically forming a light beam including infrared rays reflected from a user, and further digitizes the infrared image and then outputs a digital image signal. . The arithmetic control module is connected to an image capturing device, receives the digital image signal, recognizes a change of the user image with respect to the time axis in the plane reference coordinates, and operates / controls the signal according to various body movements of the user. Is generated. The graphical user interface is displayed on the screen and connected to the image sensor unit to receive operation / control signals and display specific output responses accordingly.

The present invention has the following remarkable effects:
First, the image sensor unit according to the present invention can accurately recognize a user's body image and has a preferable high sensitivity.

  Second, the image sensor unit further calculates the depth of field for the user's body image and obtains the user's body dimensions or information that moves back and forth, so that the user can fully Operations can be used to control / control the graphical user interface.

  The user does not need to use a separate input device, such as a keyboard, mouse, touch panel, joystick, etc. Can be operated using.

  4. The present invention can sufficiently provide various graphical user interface operations. Even if the graphical user interface appears as a planar image or as a three-dimensional scenario, it can achieve the effect of rich virtual reality.

  For a fuller understanding of the features and technical contents of the present invention, reference may be made to the following detailed description and drawings related to the invention. However, the attached drawings are only used for reference and description and do not limit the present invention.

  FIG. 1 and FIG. 2 are a structural block diagram and a usage state diagram of an embodiment of a graphical user interface (GUI) operation / control apparatus according to the present invention, respectively. The graphical user interface according to the present invention includes an image sensor unit 1 and a graphical user interface 2, and the graphical user interface 2 is connected to the image sensor unit 1 and displayed on the screen 20.

  The image sensor unit 1 includes an infrared light emitting device 11, an image capturing device 12, and a calculation control module 13. The infrared light emitting device 11 is for emitting infrared rays toward a user. The infrared light emitting device 11 includes a plurality of infrared light emitting units 111 and is, for example, a light-emitting diode that can emit infrared rays having a wavelength of 750 to 1300 nm. . The preferred infrared wavelength in this example is about 850 nm. As shown in FIG. 2, the infrared light emitting device 11 is disposed outside the image capturing device 12. The infrared light emitting unit 11 surrounds the image capturing device 12. In the present embodiment, the infrared light emitting units 11 are arranged in a ring shape, but the present invention is not limited to this example, and the infrared light emitting units 11 are arranged in a rectangular shape or a specific curve so as to be uniform for the user. It can also emit infrared rays. The infrared light emitting device 11 may be arranged on the upper side of the display screen 20 and emits infrared rays uniformly toward the user who is operating the graphical user interface 2.

  The image capturing device 12 includes an infrared filtering module 121 and an image sensor module 122. The infrared filtering module 121 includes a color filter, which filters light rays other than infrared wavelengths and allows only infrared rays to pass through. The image capturing device 12 receives an infrared image by passing a light beam including infrared rays reflected from a user by the infrared filtering module 121 and optically forming an image. The image sensor module 122 receives the infrared image to increase the contrast between the user image and the background image in the infrared image, and further digitizes the infrared image before outputting a digital image signal. The digital image signal includes a user image and a background image. In this embodiment, in order to increase the contrast between the user image and the background image, the brightness of the user image is set higher than the brightness of the background image, or the brightness of the user image is set lower than the brightness of the background image. can do. Alternatively, additional information can be provided in advance. For example, one basic value is set, and the digital image signal is localized. If the variation in the localized digital image signal is higher than the basic value, the user image (foreground) is set, and if not, the background image is set. In this way, the user's physical motion can be further recognized by capturing and locking the user image and removing the background image.

  The user and the environment background have different depths of focus because the distance to the image capture device 12 is different. Therefore, the arithmetic control module 13 is connected to the image capturing device 12, receives the digital image signal, and calculates the depth of focus of the user image in the digital image signal, thereby converting the environment background image into the digital image signal. Provide additional information removed from. Then, tracking is performed by locking only the user image, filtering an image having the same depth of focus as the user image other than the locked user image, and defining the locked user image as a plane reference coordinate. By performing arithmetic processing, the user image recognizes a change with respect to the time axis in the plane reference coordinates, and generates an operation / control signal in accordance with various movements of the user's body. As shown in FIG. 2, when two image capturing modules 12 are connected to the arithmetic control module 13, the arithmetic control module 13 can receive digital image signals from different image capturing devices 12, and the user image The depth of focus of the user is compared and calculated, and the information of the three-dimensional dimensions of the contour of the user's body can be obtained more easily, so that the user's limbs can be accurately distinguished whether they are in an overlapping state, The user can be recognized with respect to the movement or acceleration of the image sensor unit 1 or the front and back of the screen 20.

  The graphical user interface 2 receives the operation / control signal generated from the arithmetic control module 13 and displays a specific output reaction in accordance with the operation / control signal. Among them, the screen 20 may be a flat panel display or may be displayed by being projected on a screen by a projector. As shown in FIG. 3, the graphical user interface 2 is represented by a two-dimensional planar image type operation interface. For example, the user interface includes objects such as windows, icons, frames, menus, and pointers 21. Among them, the pointer 21 can perform a specific reaction according to a specific body movement of the user, for example, move up, down, left, right, select, open, and the like.

  In another embodiment, as shown in FIG. 4, the graphical user interface 2 can appear not only as a two-dimensional planar image type but also as a scenario type operating environment that is a virtual reality. The object of the graphical user interface 2 can be displayed in a three-dimensional (3-dimension) stereoscopic image format, and includes an image pointer 22 representing the whole body or a specific part of the user's body. It functions as a pointer according to the user's physical movements. For example, the image pointer 22 is displayed corresponding to the movement or movement of the user according to the movement of the user's body or the movement of the limbs. In particular, when the user moves back and forth with respect to the image sensor unit 1 or the screen 20, or particularly when the limb moves forward and backward, for example, hitting the palm forward or punching. In the graphical user interface 2, the image pointer 22 corresponding to the user's body can display a specific output reaction, for example, selection, click, open, etc., by the operation / control signal generated from the image sensor unit 1 already received. .

  In the above embodiment, the graphical user interface 2 can appear in various specific scenarios, such as a living room, a meeting, a banquet, and the like. As shown in FIG. 5, a plurality of image sensor units 1 can be further connected by a personal computer or an internet server to support a plurality of users. Multiple users can interact with the corresponding image pointers 22 in the graphical user interface 2 to enhance the user's feeling of overflowing be personally on the scene and to enrich the virtual reality. The effect can be achieved.

  In summary, the present invention has the following preferable effects.

  1. The image sensor unit 1 according to the present invention has an excellent background image filtering effect, can accurately recognize a user's body image, and has a preferable high sensitivity.

  Second, the image sensor unit 1 further performs calculation processing on the depth of focus of the user body image, and obtains information on the size of the user's body or moving back and forth, so that the user can fully perform his / her body movement. It is used to operate / control the graphical user interface.

  3. The user does not need the assistance of a separate input device, such as a keyboard, mouse, touch panel, joystick, etc., and naturally blends into the scenario that appears in the graphical user interface and operates by his / her own body movement can do.

  4. The present invention can sufficiently provide various types of graphical user interface operations, and even if the graphical user interface appears as a planar image type or a three-dimensional scenario type, a richer virtual reality effect can be obtained. Can be achieved.

  The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the claims of the present invention. Technical changes of equivalent effects made by operating the contents of the specification and drawings of the present invention are not limited thereto. These are all included in the scope of protection of the rights of the present invention, and will be described together here.

1 is a structural block diagram of an embodiment of an operation / control device of a graphical user interface according to the present invention; FIG. FIG. 2 is a usage diagram of an embodiment of an operation / control apparatus of a graphical user interface according to the present invention. FIG. 4 is a diagram representing an operation interface of a graphical user interface according to an embodiment of the present invention. It is a figure which shows the scenario type operation environment which is virtual reality by a certain embodiment of this invention. 1 is a diagram of a system in which a plurality of image sensor units are connected to correspond to a plurality of users by a personal computer or an Internet server according to an embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image sensor unit 11 Infrared light emitting device 111 Infrared light emitting unit 12 Image capturing device 121 Infrared filtering module 122 Image sensor module 13 Operation control module 2 Graphical user interface 21 Pointer 22 Icon pointer

Claims (5)

In a graphical user interface operation / control device comprising an image sensor unit and a graphical user interface,
The image sensor includes an infrared light emitting device, an image capturing device, and a calculation control module.
The infrared light emitting device is for emitting infrared rays toward the user,
The image capturing device receives an infrared image by optically forming a light beam including infrared rays reflected from a user, and outputs a digital image signal after digitizing the infrared image. ,
The arithmetic control module is connected to an image capturing device, receives the digital image signal, and recognizes a change of the user image with respect to a time axis in a plane reference coordinate, thereby operating / control signals according to various body movements of the user. Produces:
The graphical user interface is displayed on a screen and connected to the image sensor unit to receive an operation / control signal and display a specific output response accordingly. .   The arithmetic control module receives the digital image signal output from the image capturing device, calculates a depth of focus of the user image in the digital image signal, and provides additional information obtained by removing the background image from the digital image signal. 2. The operation / control apparatus of a graphical user interface according to claim 1, wherein only the user image is locked, and further, the locked user image is defined as a plane reference coordinate, and calculation processing is performed.   2. The operation / control apparatus of a graphical user interface according to claim 1, wherein the arithmetic control module filters an image corresponding to the depth of focus of the user image as well as a locked user image.   A plurality of image capturing devices, wherein the arithmetic control module is connected to the plurality of image capturing devices, receives a plurality of digital image signals from the plurality of image capturing devices, and a user image in the plurality of digital image signals; Calculating the position or spatial dimension information of a specific part of the user's body by comparing differences and changes in the depth of focus of the user, and discriminating whether the user's limbs are overlapping or not The operation / control of the graphical user interface according to claim 1, wherein an operation / control signal corresponding to a user's body movement is generated by distinguishing movements such as body distance, movement, acceleration or rotation. apparatus.   2. The graphical user interface according to claim 1, wherein the graphical user interface includes an image pointer representative of the whole body or a specific part of the user's body, and the image pointer corresponds to a user's body movement and functions as a pointer. Graphical user interface operation / control device.

Images