KR20110115508A - Method and apparatus for user interraction - Google Patents

Abstract

The present invention relates to a user interaction apparatus and method, wherein the disclosed user interaction apparatus includes a pattern frame output unit for outputting a structural image pattern frame, an image output unit for outputting an image including a structural image pattern frame, and an output of the image output unit. An image acquisition unit for capturing the structural image pattern frame from the image, and an image recognition unit for recognizing a user input by processing the structural image pattern frame captured by the image acquisition unit, the captured image while the user can interact with bare hands It requires low computational time and requires a short time for interaction processing, thereby providing fast response time, and by using pattern frame, it shows strong recognition of skin color and ambient light, and projector brightness, color, focus, etc. Quickly calibrate the output screen to a specific space There is an advantage in that it can be accurately matched without distortion.

Description

Device and method of user interaction {METHOD AND APPARATUS FOR USER INTERRACTION}

The present invention relates to a user interaction apparatus and method, and more particularly, to a user interaction apparatus and method for recognizing a user input through interaction with a user in a projection computer having a projector and a camera.

Recently, due to the development of technology, projectors and cameras have been miniaturized and mounted on mobile devices, and their use value has increased.

In addition, prototypes have been developed for making small projectors and cameras wearable, hanging on the neck or shoulders, and providing various services.

1A and 1B illustrate a usage example of a wearable projection computer according to the related art, which illustrates a projection computer system including an output means such as a projector and a camera for input.

In order to interact with the user in such a projection computer, it is very important to sense and recognize the user's input in the projected image.

For efficient interaction in a mobile environment, a method of projecting an image on a palm or a table and using a bare finger to interact is effective. In order to perform this in a mobile embedded system, a low computation type recognition technology is required.

According to the related art, in order to increase the recognition performance of the interaction through the posture recognition of the user, a method of physically wearing an interaction recognition improving means such as a color marker on the hand or the finger has been used, but the user may recognize the interaction such as the color marker. There was an uncomfortable problem of carrying the improvement means.

In order to alleviate this inconvenience, technologies for interacting with bare-hands have also been developed, but natural input captured by the camera is processed to recognize user input, in which case discerning user input from natural video. In order to require a high level of computation, interaction processing requires a long time, and in the case of an embedded system, there is a problem in that it does not provide fast response time. In particular, it is very difficult to recognize a touch operation using a bare finger on a surface such as a palm or a table, which requires a large amount of computation.

In addition, in order to provide quality service using the projector, the brightness, color, focus, etc. of the output image should be well matched, and distortion should not occur. For this purpose, the brightness, color, focus, etc. of the projector should be adjusted and output. It is necessary to exactly match the screen to a specific space. According to the related art, a high level of computation is required for this purpose, and a long time is required for correction or registration.

The present invention has been proposed to solve the above problems of the prior art, and the user can interact with the bare hand, but low computation is required for the captured image, which requires a short time in the interaction process, thereby providing a fast response time. Provided is an interaction apparatus and method.

In addition, the present invention allows to quickly correct the brightness, color, focus, etc. of the projector and to quickly and accurately match the output screen to a specific space.

As a first aspect of the present invention, a user interaction apparatus includes a pattern frame output unit configured to output a structured light pattern frame, an image output unit configured to output an image including the structured image pattern frame, and the image output unit. An image acquisition unit may capture the structural image pattern frame from an output image, and an image recognition unit configured to image the structural image pattern frame captured by the image acquisition unit to recognize a user input.

The image output unit may include an additional device or a projector for outputting the structural image pattern frame.

Here, the structure image pattern frame may use a monochrome pattern frame or a color pattern frame.

The structure image pattern frame may use an infrared pattern frame, and the user interaction device may further include a light source controller configured to drive a red / cyan / infrared light source to output an image including the infrared pattern frame.

The structure image pattern frame may include a grid shape.

The image recognition unit may recognize at least one or more states of a focus state or a distortion state of the structural image pattern frame processed by the image, and the user interaction apparatus may include an optical system included in the image output unit according to the recognized state. The optical control unit may further include controlling an at least one of focus, pan, or tilt of the optical control unit to correct at least one of focus or distortion.

The image recognition unit may recognize at least one or more states of the brightness state or the color state of the structural image pattern frame processed by the image, and the image output unit may include at least one of the brightness state or the color state according to the recognized state. Can be corrected.

According to a second aspect of the present invention, a user interaction method of a user interaction apparatus including an image output unit and an image acquisition unit includes outputting an image including a structure image pattern frame through the image output unit, and using the image acquisition unit. The method may include capturing the structural image pattern frame from the output image of the image output unit and recognizing a user input by image processing the captured structural image pattern frame.

Here, the structure image pattern frame may use a monochrome pattern frame or a color pattern frame.

The structure image pattern frame may use an infrared pattern frame, and the user interaction method may further include driving a red / cyan / infrared light source to output an image including the infrared pattern frame.

The structure image pattern frame may include a grid shape.

The recognizing of the user input may include recognizing at least one of a focus state or a distorted state of the structured image pattern frame processed by the image, and the user interaction method includes: the image output unit according to the recognized state; The method may further include correcting at least one of focus or distortion by controlling at least one of focus, pan, or tilt of the optical system included in the optical system.

The recognizing of the user input may include recognizing at least one of a brightness state or a color state of the structured image pattern frame processed by the image, and the user interaction method may include a brightness state or a color according to the recognized state. The method may further include correcting at least one of the states.

The recognizing of the user input may extract the structural image pattern frame from the output image of the image output unit captured by the image acquisition unit, and then recognize the user input by recognizing a change in shape or brightness.

According to an embodiment of the present invention, while the user can interact with the bare hand, low computation is required for the captured image, which requires a short time for the interaction process, thereby providing a fast response time.

And by using pattern frame, it shows strong recognition of skin color and ambient light.

You can also quickly calibrate the projector's brightness, color, and focus, and align the output screen quickly and accurately without distortion in a specific space.

The present invention can be used not only for a projector system, such as a projection computer, but also for a mobile device having an image output device and an image acquisition device, and in particular, a mobile device or a wearable system in which changes in the surrounding environment such as ambient light quantity, lighting, and shaking are severe. It is also more valuable in small embedded systems that require low computational technology.

1A and 1B show an example of use of a projection computer according to the prior art.
2 is a block diagram of a user interaction apparatus according to an embodiment of the present invention.
3 and 4 are exemplary diagrams of a structural image pattern frame according to an embodiment of the present invention.
5 is a view showing an insertion state and a capture state of the structure image pattern frame according to an embodiment of the present invention.
6 is an example of a structure image pattern frame captured by a user interaction device according to an embodiment of the present invention.
7 is a view showing a state in which only a pattern extracted from the image of FIG.
8 illustrates distortion of an output image pattern due to non-planar or tilted projectiles.
9 is a view showing a structure image pattern frame captured when the focus is out of focus.
10 is an exemplary view of a projector / camera integrated projection system according to an embodiment of the present invention.
11 is a diagram illustrating a state in which an RGB / IR light source is added to a DLP type projector system.
12 to 14 are configuration diagrams of a projector for outputting a structured image pattern frame according to an embodiment of the present invention.
15 is a flowchart illustrating a user interaction method according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms, and only the embodiments of the present invention make the disclosure of the present invention complete and the general knowledge in the technical field to which the present invention belongs. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

Combinations of each block of the accompanying block diagram and each step of the flowchart may be performed by computer program instructions. These computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment such that instructions executed through the processor of the computer or other programmable data processing equipment may not be included in each block or flowchart of the block diagram. It will create means for performing the functions described in each step. These computer program instructions may be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, and thus the computer usable or computer readable memory. It is also possible for the instructions stored in to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of each step of the block diagram. Computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to create a computer or other programmable data. Instructions that perform processing equipment may also provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions noted in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

2 is a block diagram of a user interaction apparatus according to an exemplary embodiment of the present invention.

As shown in the drawing, the user interaction apparatus includes a pattern frame output unit 100, an image output unit 200, a light source control unit 300, an optical control unit 400, an image acquisition unit 500, a synchronization unit 600, and an image. The recognition unit 700 may be included, and the light source control unit 300, the optical control unit 400, and the synchronization unit 600 may be omitted depending on the implementation.

The pattern frame output unit 100 outputs a structural image pattern frame. Here, the structure image pattern frame may use a monochrome pattern frame, a color pattern frame, or an infrared pattern frame, and may include a grid shape.

The image output unit 200 projects an image including a structural image pattern frame onto a display area such as a screen. When the image recognizer 700 recognizes the brightness state and / or the color state of the pattern frame captured by the image acquisition unit 500, the image output unit 200 may display the brightness state and / or the color state according to the recognized state. Can be corrected.

The synchronizer 600 synchronizes the frames of the image output unit 200 and the image acquirer 500.

The image acquisition unit 500 captures a structural image pattern frame from the output image of the image output unit 200. Here, when the pattern frame output unit 100 adds an infrared pattern frame to the input image, the image acquisition unit 500 should have an infrared camera function.

The image recognizer 700 recognizes a user input by image processing the structural image pattern frame captured by the image acquirer 500. Here, when the pattern frame output unit 100 adds a monochromatic pattern frame or a color pattern frame to the input image, the image recognition unit 700 may be configured to select from among the brightness state, color state, focus state, or distortion state of the patterned pattern frame. Recognize at least one or all states or all states.

The optical controller 400 is included in the image output unit 200 according to the recognized state when the image recognizer 700 recognizes at least one or all or all states of the focus state or the distortion state of the structural image pattern frame. At least one or all or all of the focus, pan, or tilt of the optical system is controlled to correct at least one or all or all of the focus or distortion.

The light source controller 300 is a component required when the pattern frame output unit 100 adds an infrared pattern frame to an input image. The light source control unit 300 adds an infrared pattern frame to the input image of the image output unit 200. Drive the infrared light source.

3 and 4 are exemplary views of a pattern frame according to an embodiment of the present invention.

The structure image pattern frame added by the pattern frame output unit 100 to the input image of the image output unit 200 may be implemented in various forms including a grid form, and FIG. 3 and FIG. Will be shown. As shown in the structure image pattern frame of FIG. 3, a start notification pattern 111, an end notification pattern 112, an RGB color correction pattern 113, a horizontal column, a vertical column, and an index pattern 114 may be included. As shown in Fig. 4, it may be implemented in the form of a color adjustment screen of the broadcast signal. In particular, AR technology can be easily combined with a marker-type pattern, which is a method commonly used in existing AR (Augmented Reality) technology.

The structure image pattern frame is projected to the output image by the image output unit 200 and is captured by the image acquisition unit 500 synchronized with the image output unit 200.

FIG. 5 illustrates a method of capturing a structural image pattern frame to which a structure image pattern frame is added to an output image and a synchronized image acquisition unit. The structural image pattern frame may be inserted between normal image frames according to a preset period, and the image output unit 500 in which the image output unit 200 and the frame are synchronized captures the structural image pattern frame according to the preset period. .

As the structure image pattern frame, a monochrome pattern frame, a color pattern frame, an infrared pattern frame, etc. may be used depending on the purpose. Monochromatic and color pattern frames can be used for color and brightness correction, and most projector modules that can be implemented in the embodiment of the image output unit 200 can be used. In addition, there is an advantage that can be inserted into the pattern of the color that is easy to distinguish from the background. However, for the quality of the output screen, since the structural image pattern frame should not be visible to the user, there is a limit to the number of structural image pattern frames that can be inserted per unit time. In order to insert a frame, high speed pattern insertion and high speed screen capture are required. Infrared pattern frames can only obtain pattern images when capturing images with infrared cameras, making image processing easier and are almost invisible to the human eye, so there is almost no limit to the number of structured image pattern frames that can be inserted per hour. However, it cannot be used for color and brightness correction and can be used in projector modules that use infrared light as a light source as well as red-cyan light sources.

When the structure image pattern frame is used as in the embodiment of the present invention, since the time for outputting the pattern image is different from the general image output from the image output unit 200, the image acquisition unit 500 only at the time when the structure image pattern frame is output. Importing an image from an image and performing image processing does not affect the user's interaction recognition because it can recognize user input.

FIG. 6 is an example of a pattern frame captured by a user interaction apparatus according to an exemplary embodiment of the present invention, and illustrates a changed shape of a finger touch gesture and a metric gesture, and FIG. 7 extracts only a pattern from the image of FIG. It is a figure showing the state.

In FIG. 6, (a) is an image captured by the image acquisition unit 500 during a finger touch operation, and (b) is an image captured by the image acquisition unit 500 during a finger metric operation. When a user touches a surface such as a palm or a table with his or her finger, the finger and the surface are located on the almost same plane, so if only the pattern is extracted from the image recognition unit 700, as shown in FIG. The pattern change (distortion, brightness, thickness, etc.) at the tip of the finger is hardly generated, and when the user does not touch the surface with the finger, only the pattern is recognized by the image recognition unit 700 due to the difference in perspective between the finger and the surface. When extracted, a pattern change occurs as shown in FIG.

As such, the image recognizer 700 recognizes the shape change of the pattern from the image captured by the image acquirer 500, obtains coordinate information of the converted pattern, and extracts the end point of the finger. Touch / drag / release can be recognized.

Conventionally, when the shape of a hand or a finger is recognized from an image acquired by a camera or the like, the recognition rate varies considerably according to the color of the skin or the surrounding environment. Because it recognizes the shape of the hand or finger, it is less affected by the skin color or the surrounding environment, thereby reducing the variation of the recognition rate, thereby providing a stable recognition result.

8 is a diagram illustrating distortion of an output image pattern due to a non-planar or tilted projectile. Since a pattern is extracted from an image acquired by the image acquisition unit 500 to know a distortion form generated in a projection such as a screen, the projection may be obtained as shown in FIGS. 8A and 8B. Can easily recognize and correct distortion.

9 is a view showing a state of the pattern frame captured when the focus is out of focus. When the image is not in focus, when the image acquisition unit 500 extracts a pattern from the acquired image, an unclear pattern is extracted as illustrated in FIG. 9, and the focus is corrected until a clear pattern is extracted from the structure image pattern frame. You can project a clear general image. Like the focus correction process, the brightness or the color of the output image can be easily corrected.

Hereinafter, embodiments of implementing the image output unit as a projector and implementing the image acquisition unit as a camera will be described with reference to FIGS. 10 to 14.

FIG. 10 is an exemplary diagram of a projector / camera integrated projection system according to an exemplary embodiment of the present invention, which illustrates a case of employing a mobile device.

In FIG. 10A, 201 is a projector, 301 is a projector and a camera control module, which corresponds to the pattern frame output unit 100 and the synchronization unit 600 of FIG. 2, and 401 is a focus control motor. Corresponds to the controller 400, 501 is a general camera.

In FIG. 10 (b), 203 is a projector, 303a is a projector and a camera control module, and corresponds to the pattern frame output unit 100 and the synchronization unit 600 of FIG. 2, and 303b is a red cyan / infrared light source control module. It corresponds to the light source control unit 300 of FIG. 2, 403 corresponds to the optical control unit 400 of FIG. 2 as a focus control motor, 503a is a general camera, and 503b is an infrared camera.

In the embodiment of FIG. 10A, a monochromatic pattern frame or a color pattern frame may be used. In the embodiment of FIG. 10B, a monochromatic pattern frame, a color pattern frame, or an infrared pattern frame may be used.

FIG. 11 is a diagram illustrating a state in which a red cyan / infrared light source is added to a DLP (Digital Light Processing) type projector system. In order to use the infrared pattern frame according to an embodiment of the present invention, the red-green / infrared light source 205a must be added so that the DLP projector unit 205 can provide an output image including the infrared pattern frame. The light source controller 305 for driving the infrared light source 205a may be driven by an infrared enable signal applied according to the insertion of the infrared pattern frame.

12 to 14 are configuration diagrams of a projector for inserting a pattern frame according to an embodiment of the present invention, Figure 12 shows a DLP type projector 210 including a red-cyan / infrared integrated light source 211, FIG. 13 illustrates a DLP type projector 220 in which a red cyan light source 221a and an infrared light source 221b are separated, and a three liquid crystal (3LCD) type projector 230 including a red cyan / infrared light source 231. ).

In FIG. 12, 213a is a blue pass filter, 213b is a green pass filter, 213c is a red pass filter, 213d is an infrared pass filter, 215 is a digital micromirror device (DMD), 217 is a projection lens, and 219 is a screen to be.

In Figure 13, 222 is a prism, 223a is a blue pass filter, 223b is a green pass filter, 223c is a red pass filter, 223d is an infrared pass filter, 225 is a DMD, 227 is a projection lens, 229 is a screen.

In FIG. 14, 223a, 223b, and 223c are dichroic mirrors, 235a, 235b, 235c, 235d, 235e, and 235f are mirrors, 237 are prisms, 237 are projection lenses, and 239 are screens.

15 is a flowchart illustrating a user interaction method according to an exemplary embodiment of the present invention.

Referring to FIGS. 2 and 15, the user interaction method by the user interaction apparatus according to the embodiment of the present invention will be described in time series.

First, in the system initialization state, the synchronization unit 600 synchronizes the frames of the image output unit 200 and the image acquisition unit 500 (S801).

In addition, the pattern frame output unit 100 adds the structure image pattern frame at predetermined intervals to the input image of the image output unit 200 (S845 and S847), and the image output unit 200 outputs a general image. The structure image pattern frame is output as an image for each cycle (S849). In FIG. 15, since the flow is shown based on the situation in which the structure image pattern frame is added by the previous process, step S803 is included, and step S845 is positioned after the step S843.

The image acquisition unit 500 acquires an image by capturing an output image of the image output unit 200, that is, a structural image pattern frame at a point in time when the structural image pattern frame is output by the image output unit 200 (S811).

Next, the image recognition unit 700 recognizes a user input by performing image processing (S821 and S831) on the structure image pattern frame captured by the image acquisition unit 500 (S835). For example, after extracting the structural image pattern frame from the output image of the image output unit 200 captured by the image acquisition unit 500, the shape and / or brightness change may be recognized to recognize a user input.

In FIG. 15, the steps S813 and S815 are performed prior to the image processing because the brightness or the color state cannot be recognized in the case of the infrared pattern frame. Thus, in the case of the monochrome pattern frame or the color pattern frame, the image recognition unit 700 recognizes the brightness and / or color state of the pattern frame and provides the result to the image output unit 200 (S833).

In addition, the image recognition unit 700 recognizes a focus state and / or a distortion state of the pattern frame image-processed in step S821 or S831 and transmits the image to the optical control unit 400 (S837 and S839).

Then, the image acquisition unit 200 corrects the brightness state and / or the color state according to the brightness and / or the color state recognized by the image recognition unit 700 (S841), and the optical controller 400 performs the image recognition unit. The focus and / or distortion is corrected by controlling at least one or more of the focus, pan, or tilt of the optical system included in the image output unit 200 according to the focus and / or distortion state recognized by the 700 (S842 and S843). ).

100: pattern frame output unit 200: image output unit
205: DLP projector unit 205a: red cyan / infrared light source
300, 305: light source control unit 400: optical control unit
500: image acquisition unit 600: synchronization unit
700: image recognition unit

Claims (13)

A pattern frame output unit for outputting a structured light pattern frame,
An image output unit configured to output an image including the structural image pattern frame;
An image acquisition unit for capturing the structural image pattern frame from the output image of the image output unit;
And an image recognition unit configured to image the structure image pattern frame captured by the image acquisition unit to recognize a user input.
User Interaction Device. The method of claim 1,
The structured image pattern frame uses a monochrome pattern frame or a color pattern frame.
User Interaction Device. The method of claim 1,
The structure image pattern frame uses an infrared pattern frame,
The user interaction apparatus may further include a light source controller configured to drive a red cyan / infrared light source to output an image including the infrared pattern frame.
User Interaction Device. The method according to any one of claims 1 to 3,
The structure image pattern frame includes a grid shape.
User Interaction Device. The method according to any one of claims 1 to 3,
The image recognition unit recognizes at least one or more of a focus state or a distortion state of the structural image pattern frame processed by the image,
The user interaction apparatus may further include an optical controller configured to correct at least one or more of focus or distortion by controlling at least one or more of focus, pan, or tilt of an optical system included in the image output unit according to the recognized state.
User Interaction Device. The method according to claim 1 or 2,
The image recognition unit recognizes at least one or more of the brightness state or the color state of the structural image pattern frame processed by the image,
The image output unit corrects at least one of a brightness state or a color state according to the recognized state.
User Interaction Device. A user interaction method of a user interaction apparatus including an image output unit and an image acquisition unit,
Outputting an image including a structural image pattern frame through the image output unit;
Capturing the structural image pattern frame from the output image of the image output unit by using the image acquisition unit;
And recognizing a user input by image processing the captured structural image pattern frame.
User interaction method. The method of claim 7, wherein
The structured image pattern frame uses a monochrome pattern frame or a color pattern frame.
User interaction method. The method of claim 7, wherein
The structure image pattern frame uses an infrared pattern frame,
The user interaction method may further include driving a red / cyan / infrared light source to output an image including the infrared pattern frame.
User interaction method. The method according to any one of claims 7 to 9,
The structure image pattern frame includes a grid shape.
User interaction method. The method according to any one of claims 7 to 9,
The recognizing of the user input may include recognizing at least one of a focus state and a distorted state of the structural image pattern frame processed by the image.
The user interaction method may further include correcting at least one or more of focus or distortion by controlling at least one or more of focus, pan, or tilt of an optical system included in the image output unit according to the recognized state.
User interaction method. The method according to claim 7 or 8,
The recognizing of the user input may include recognizing at least one of a brightness state or a color state of the structural image pattern frame processed by the image.
The user interaction method further comprises correcting at least one of a brightness state or a color state according to the recognized state.
User interaction method. The method according to claim 6,
The recognizing of the user input may include extracting the structural image pattern frame from the output image of the image output unit captured by the image acquisition unit, and recognizing the user input by recognizing a change in shape or brightness.
User interaction method.

Images