KR102156175B1 - Interfacing device of providing user interface expoliting multi-modality and mehod thereof - Google Patents

Abstract

Disclosed is an interfacing device that provides a user interface utilizing multi-modality. An embodiment recognizes at least two modality inputs for controlling a scene, and generates scene control information based on the at least two modality inputs.

Description

An interfacing device that provides a user interface utilizing multi-modality, and a method using the device {INTERFACING DEVICE OF PROVIDING USER INTERFACE EXPOLITING MULTI-MODALITY AND MEHOD THEREOF}

The following embodiments relate to an interfacing device that provides a user interface utilizing multi-modality and a method using the device.

Human sensing technology can provide an intuitive user interface. Such human sensing technology is being applied to various home appliances.

In general, the human sensing technology is implemented to perform an action corresponding to the detected motion when a predetermined specific motion is detected in order to manipulate the home appliance.

An interfacing device that provides a user interface utilizing multi-modality according to one side includes: a scene parameter acquisition unit acquiring a scene parameter including information related to a scene; A multi-modality recognition unit for recognizing at least two modality inputs for controlling the scene; And a scene control information generator that generates scene control information based on the scene parameter and the at least two modality inputs.

In this case, the interfacing device further includes a user customization parameter obtaining unit that obtains a user customization parameter, and the new control information generation unit analyzes a combination of the at least two modality inputs based on the user customization parameter. It may include an analysis unit.

Here, the user-customized parameter corresponds to a specific user, and selects each of a plurality of predetermined control aspects of the scene from one of a plurality of modality inputs or two or more modality inputs according to the preference of the specific user. Mapping information that maps in combination may be included.

In addition, the interfacing device may further include a user identification unit for identifying a user, and the user customized parameter obtaining unit may obtain a user customized parameter corresponding to the identified user.

The interfacing method for providing a user interface utilizing multi-modality according to the other side includes: obtaining a scene parameter including information related to a scene; Recognizing at least two modality inputs for controlling the scene; And generating scene control information based on the scene parameter and the at least two modality inputs.

1 is a block diagram illustrating an interfacing device providing a user interface utilizing multi-modality according to an embodiment.
2A to 2C are views for explaining a user interface utilizing multi-modality according to an embodiment.
3A to 3B are block diagrams illustrating an interfacing device using a user-customized parameter according to an embodiment.
4 is a view for explaining an interfacing system that provides a user interface utilizing multi-modality according to an embodiment.
5 is a flowchart illustrating an interfacing method for providing a user interface utilizing multi-modality according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an interfacing device that provides a user interface utilizing multi-modality according to an embodiment.

Referring to FIG. 1, an interfacing device 100 according to an embodiment provides a user interface (UI) using multi modalities.

Here, modality is a primary sensory aspect such as sight or hearing, and multi-modality is a technique using a combination of at least two different types of modalities.

The interfacing apparatus 100 according to an embodiment may include a scene parameter acquisition unit 110, a multi-modality recognition unit 120, and a scene control information generation unit.

The new parameter acquisition unit 110 may acquire a new parameter. Here, the scene is multimedia content provided to the user by the scene expression device 150 and may comprehensively include a flat image, a stereoscopic image, and sound.

The scene parameter may include information related to a scene, and in particular, may include information related to a characteristic of a scene capable of interacting with a user.

For example, the scene parameter may include a parameter related to the size of at least one object included in the scene or a parameter related to the location of the corresponding object. Each of these at least one object may be controlled through a user interface provided by the interfacing device 100.

In addition, the new parameter may include a parameter related to movement control of a corresponding object, a parameter related to rotation control of a corresponding object, or a parameter related to scale control of the corresponding object. For example, the scene parameter may include information indicating whether the object can be moved, whether it can be rotated, or whether it can be enlarged or reduced according to a user's input.

Also, the scene parameter may include a parameter related to the viewpoint of the scene. For example, the scene parameter may include information related to the viewpoint of the currently displayed scene, information on whether the viewpoint of the scene can be changed, or a list of viewpoints of the scene that can be changed.

The multi-modality recognition unit 120 may recognize at least two modality inputs for controlling a scene.

Although not shown in the drawing, the multi-modality recognition unit 120 may receive sensor data from a sensor and recognize at least two modality inputs based on the received sensor data. Here, the sensor may include an image sensor that photographs an object, a depth sensor that measures a distance to the object, or a microphone that acquires voice data.

The multi-modality recognition unit 120 may perform an algorithm for recognizing a multi-modality input by using the received sensor data. According to an embodiment, the multi-modality recognition unit 120 may be implemented as a processor on which a program in which a corresponding algorithm is described is executed, or a System on Chip (SoC) composed of modules for performing the algorithm.

The multi-modality recognition unit 120 includes at least one of the gaze recognition unit 121, the fingertip recognition unit 122, the hand posture recognition unit 123, the hand motion recognition unit 124, or the voice recognition unit 125. Can include.

Here, the gaze recognition unit 121 may recognize the gaze of the user. In some cases, the gaze recognition unit 121 may recognize the direction of the user's gaze or a position in the scene where the gaze of the user is directed. The gaze recognition unit 121 may recognize a user's gaze by processing image data received from an image sensor or depth data received from a depth sensor. In addition, the gaze recognition unit 121 may detect the user's eye position or the user's eye orientation. In this case, the gaze recognition unit 121 may use relative coordinates based on the interfacing device 100 or absolute coordinates indicating a three-dimensional space in which the user is located.

The fingertip recognition unit 122 may recognize the user's fingertip. Likewise, the fingertip recognition unit 122 may recognize a direction indicated by the user's fingertip or a position indicated by the user's fingertip within a scene. The fingertip recognition unit 122 may recognize the user's fingertip by processing image data received from the image sensor or depth data received from the depth sensor. In addition, the fingertip recognition unit 122 may detect the positions of both hands of the user and the positions of ten fingertips. In this case, the fingertip recognition unit 122 may use relative coordinates based on the interfacing device 100 or absolute coordinates indicating a three-dimensional space in which the user is located.

The hand posture recognition unit 123 may recognize the user's hand posture. For example, the hand posture recognition unit 123 may recognize whether the user's hand is in the scissors posture, rock posture, or paper posture. The hand posture recognition unit 123 may recognize the user's hand posture by processing image data received from the image sensor or depth data received from the depth sensor. Alternatively, the hand posture recognition unit 123 can recognize whether the user's hand is in a posture for pointing with only the index finger open, and whether the user's hand is in a posture as if holding a pen using the thumb and forefinger. have.

In this case, the hand posture recognition unit 123 may recognize the user's hand posture using a hand posture database provided in advance. For example, the hand posture recognition unit 123 may recognize the user's hand posture by selecting a hand posture that is most similar to the detected hand posture from among a plurality of hand postures stored in the hand posture database. In this case, the hand posture recognition unit 123 may output the selected hand posture type.

The hand motion recognition unit 124 may recognize a user's hand motion. For example, the hand motion recognition unit 124 may recognize a user's hand motion pattern. In some cases, the hand motion recognition unit 124 may perform a motion in which the user's hand rotates around the wrist, a motion in which the user's hand makes a circular motion based on a specific point in the three-dimensional space, and the user's hand in a three-dimensional space. You can recognize moving movements, etc. The hand motion recognition unit 124 may recognize a user's hand motion by processing image data received from the image sensor or depth data received from the depth sensor.

In this case, the hand motion recognition unit 124 may recognize a user's hand motion using a hand motion database provided in advance. For example, the hand motion recognition unit 124 may recognize the user's hand motion by selecting a hand motion most similar to the detected user's hand motion among a plurality of hand motions stored in the hand motion database. In this case, the hand motion recognition unit 124 may output the selected hand motion type.

The voice recognition unit 125 may recognize a user's voice. In this case, the voice recognition unit 125 may recognize the user's voice and output the recognized content in the form of a text string. The voice recognition unit 125 may recognize the user's voice by processing voice data received from the microphone.

The scene control information generation unit 130 may generate scene control information based on a scene parameter obtained by the scene parameter acquisition unit 110 and at least two modality inputs recognized by the multi-modality recognition unit 120. have.

Here, the scene control information is a control command specific to a scene, and may include information for controlling an object included in the scene or a viewpoint of the scene itself.

For example, the scene control information may include a location element including information related to a specific location in the scene. The scene expression device 150 may specify a target object controlled by the user by using the location element included in the scene control information. In some cases, the location element may be expressed in 3D coordinates or 2D coordinates.

In addition, the scene control information may include a transformation element including information related to transformation of an object located at a specific position indicated by the position element. The scene expression device 150 may control transformation of the target object specified by the location element by using the transformation element included in the scene control information.

In this case, the new expression device 150 may move the target object, rotate the target object, or enlarge or reduce the target object based on the transforming element. More specifically, when the transforming element includes a command to move the target object to another point in the scene, the scene expression device 150 may move the target object to the corresponding point. Alternatively, when the transforming element includes a command for scaling the target object by 2 times, the new expression device 150 may enlarge the target object by 2 times.

In addition, the scene control information may include a selection element including information related to selection of an object located at a specific location indicated by the location element. The scene expression device 150 may control whether to select the target object specified by the location element by using the selection element included in the scene control information. For example, when the transforming element includes an indicator for selecting a target object, the scene expression device 150 controls the scene so that the target object is selected, and otherwise, the scene expression device 150 displays the scene so that the target object is not selected. Can be controlled.

In addition, the scene control information may include a viewpoint element including information related to the viewpoint of the scene. The scene expression device 150 may change the viewpoint of the scene itself by using the viewpoint element included in the scene control information.

In addition, the new control information may include a user identification element including information related to an identifier for identifying the user. The scene expression device 150 may identify a user who wants to control the scene by using the user identification element included in the scene control information.

In addition, the scene control information may include an associated point element including information related to a plurality of points associated with a specific location indicated by the location element. The scene expression device 150 may assist an operation of controlling a target object by additionally using an associated point element included in the scene control information.

In some cases, the scene control information may further include information related to an effective pointing resolution. The scene representation device 150 may assist in controlling the pointing of an object in the scene by using information related to the effective pointing resolution.

The scene control information may be configured in an Advanced UI Device Data format as shown in Table 1 for a scene-specific command.

AUIDataFrame [[
SFVec3f position
SFNode Transform3D
SFBool Select
SFNode Viewpoint
SFString UserID
MFVec3f AssociatedPoints[N]
]]

Here, the position element may be a single field 3 ^rd order vector w/floating point representation (SFVec3f) type that expresses 3D screen coordinates, pixels, or meters specified in BifsConfig, which is a binary format for scene (BIFS) setting. .

In addition, the transform element (Transform3D) may be a single field node (SFNode) type expressing a geometric transformation intended by a user.

In addition, the selection element (Select) may be a single field Boolean (SFBool) type indicating true when a scene object specified by a location element is selected by a user.

In addition, the viewpoint element (Viewpoint) may be a single field node (SFNode) type representing the viewpoint of the scene intended by the user.

In addition, the user identification element (UserID) may be a single field string (SFString) type representing a string variable identifying each user.

In addition, the associated point elements may be of MFVec3f (Multiple Fields 3rd order Vectors w/floating point representation) type representing a set of additional points associated with a location element.

2A to 2C are diagrams for explaining a user interface utilizing multi-modality according to an exemplary embodiment.

Referring to FIG. 2A, the interfacing device 220 according to an embodiment may perform pointing control using a gaze modality.

For example, the interfacing device 220 may determine the object 221 that the user 210 wants to point from among a plurality of objects being displayed by detecting the gaze of the user 210.

Furthermore, the interfacing device 220 detects an action of continuously watching the object 221 pointed to by the user 210 for a predetermined time (eg, 3 seconds) or an action of rapidly blinking eyes twice. , It is possible to control the scene so that the object 221 being pointed is selected.

Of course, the above-described operations are only exemplary, and the configuration of the multi-modality that can be utilized by the interfacing device 220 is not limited.

Referring to FIG. 2B, the interfacing device 220 according to an embodiment may perform transformation control using both hands modality.

For example, the interfacing device 220 may move, rotate, enlarge or reduce the pointed object 221 by recognizing a motion of the user's hands 215 moving in a 3D space.

The user can perform an action such as grabbing an object with one hand, and the interfacing device 220 can select the pointed object 221 by recognizing this action.

Alternatively, the user may perform an action such as grasping an object with one hand, and then move the hand in a specific direction. In this case, the interfacing device 220 may move the pointed object 221 in a corresponding direction by recognizing these operations.

Alternatively, the user may perform an operation such as rotating the vehicle steering wheel with both hands, and the interfacing device 220 may rotate the pointed object 221 by recognizing this operation.

Alternatively, the user may perform an operation of rotating one hand around the wrist, and the interfacing device 220 may rotate the pointed object 221 by recognizing this operation.

Alternatively, the user may take an action to narrow or widen the gap between both hands while facing the palms of both hands, and the interfacing device 220 may reduce or enlarge the pointed object 221 by recognizing such an action. Can

Of course, the above-described operations are only exemplary, and the configuration of the multi-modality that can be utilized by the interfacing device 220 is not limited.

In this case, the interfacing device 220 may use both hand modality and gaze modality of FIG. 2A together.

For example, the interfacing device 220 detects the gaze of the user 210 through the gaze modality of FIG. 2A and points to the object 221, and recognizes the movement of both hands of the user through the gaze modality of FIG. It is possible to control the selection, movement, rotation, enlargement or reduction of the 221.

Of course, the above-described operations are only exemplary, and the configuration of the multi-modality that can be utilized by the interfacing device 220 is not limited.

Referring to FIG. 2C, the interfacing device 220 according to an embodiment may perform transformation control by using both a gaze modality and a voice modality.

For example, the interfacing device 220 detects the gaze of the user 210 through the gaze modality, selects the object 221, and recognizes the voice of the user 210 through the gaze modality, You can control selection, movement, rotation, zoom in or out.

Of course, the operations described above with reference to FIGS. 2A to 2C are merely exemplary, and the configuration of the multi-modality that can be utilized by the interfacing device 220 is not limited.

3A to 3B are block diagrams illustrating an interfacing device using a user-customized parameter according to an exemplary embodiment.

Referring to FIG. 3A, the interfacing device 300 according to an embodiment may further include a user customization parameter acquiring unit 340. Here, the matters described through FIG. 1 may be applied to the new parameter acquisition unit 310, the multi-modality recognition unit 320, and the scene control information generation unit 330 as they are.

The user-customized parameter acquisition unit 340 may acquire a user-customized parameter.

Here, the user-customized parameter may include mapping information for mapping a plurality of predetermined control aspects of a scene to modalities according to the user's preference in response to a specific user.

Referring to Table 2, control aspects of a plurality of predetermined scenes may include pointing, selection, movement, rotation, enlargement, reduction, and viewpoint change.

For example, assume that the interfacing device 300 acquires a first user customized parameter corresponding to user A. In this case, the first user-customized parameter may include mapping information for controlling the scene with a modality or a combination of modalities preferred by user A.

More specifically, the first user-customized parameter uses gaze modality for pointing control of objects included in scenes, eye blink modality for selection control, and movement control, rotation control, scaling control, and viewpoint change control. For this, mapping information indicating that the voice modality is used may be included.

The first user customized parameter may be set in advance by user A, and the interfacing device 300 may provide a user interface specialized for user A by acquiring such a parameter specific to user A.

In addition, it is assumed that the interfacing device 300 acquires a second user customized parameter corresponding to the user B. In this case, the second user customized parameter may include mapping information for controlling the scene with a modality or a combination of modalities favored by the user B.

In this case, the second user customized parameter may include mapping information different from the first user customized parameter. For example, the second user-customized parameter uses a fingertip modality for pointing control of an object included in a scene, a hand motion modality such as hand grip for selection control, and a hand movement for movement control, etc. Hand motion modality is used, hand motion modality such as hand rotation is used for rotational motion, voice modality is used for scaling control, and the relative position of the user's head relative to the screen is used for viewpoint change control. Mapping information indicating that location modality is used may be included.

Similarly, the second user-customized parameter may be set in advance by the user B, and the interfacing device 300 may provide a user interface specific to the user B by acquiring such a parameter specific to the user B.

The new control information generation unit 330 may analyze at least two modality inputs recognized by the multi-modality recognition unit 320 on the basis of the user-customized parameter obtained by the user-customized parameter obtaining unit 340.

For this reason, even if the same modality inputs are recognized, the interfacing device 300 according to an embodiment may generate different scene control information according to a user-customized parameter.

Although not shown in the drawings, the interfacing device 300 according to another embodiment may further include an identification unit.

In this case, the identification unit may identify a user who uses the interfacing device 300. The manner in which the identification unit identifies the user may be implemented in various ways. For example, the identification unit may identify the user by using the user's login information. Alternatively, the identification unit may identify the user by using biometric information such as the user's face, iris, and fingerprint.

The user-customized parameter obtaining unit 340 may obtain a user-customized parameter corresponding to the user identified by the identification unit. In some cases, the user-customized parameter acquisition unit 340 may obtain a user-customized parameter from a server accessible through a wired network or a wireless network.

For this reason, under a system in which the interfacing device 300 according to an exemplary embodiment is used, users can use their own interfacing settings in the same manner in various environments.

For example, user C may use a first interfacing device for a presentation in a company, a second interfacing device for watching a movie at home, and a third interfacing device for navigation operation in a car. .

According to an embodiment, the first interfacing device, the second interfacing device, and the third interfacing device may each identify user C and use a user setting parameter specialized for user C. Accordingly, user C can operate various interfacing devices in the same manner by using an input method optimized for him.

Accordingly, the interfacing device according to an embodiment may provide a technology for customizing a user interface operation preferred by a user and automatically generating commands necessary for various scenes. As a result, the user can perform a desired operation by taking an operation familiar to the user without having to recognize the type of operation required each time in various scenes or various environments.

To this end, referring to FIG. 3B, the scene control information generation unit 330 according to an embodiment may include a modality selector.

The modality selection unit may select a pointing modality, a transition modality, a rotation modality, a scaling modality, and a select modality, using the user-customized parameter obtained by the user-customized parameter acquisition unit 340.

Furthermore, the scene control information generation unit may interpret modality inputs according to the modality for pointing, modality for transition, modality for rotation, modality for scaling, and modality for select selected by the modality selector.

4 is a diagram for explaining an interfacing system that provides a user interface utilizing multi-modality according to an embodiment.

Referring to FIG. 4, the interfacing device 410 according to an embodiment may provide a user interface between the new expression device 420 and the user 430. Hereinafter, it is assumed that the interfacing device 410 is a smart phone of the user 430.

For example, the interfacing device 410 may include a first camera 411, a second camera 413, a Bluetooth transceiver 412, a microphone 414, an Internet communication unit 415, and a control unit 416. have.

The interfacing device 410 may determine the location of the new expression device 420 using the first camera 411 and may determine the location of the user 430 using the second camera 412. Furthermore, the interfacing device 410 may calculate a relative position between the scene expression device 420 and the user 430.

In some cases, the interfacing device 410 may recognize the location of the user's 430 face using the second camera 412 and transmit the information to the new expression device 420. Based on this information, the new expression device 420 may perform a paralex barrier operation or the like for providing a stereoscopic image to the user 430.

In this case, the interfacing device 410 may exchange data with the new presentation device 420 using the Bluetooth transceiver 412. Of course, the interfacing device 410 may use various near-field communication techniques in addition to the Bluetooth technique. Although not shown in the drawings, the interfacing device 410 according to another embodiment may exchange data with the new expression device 420 through a clouding environment.

In addition, the interfacing device 410 may recognize the gaze of the user 430 using the second camera 413 or may recognize a hand motion of the user 430. The interfacing device 410 may recognize the voice of the user 430 using the microphone 414.

At this time, the interfacing device 410 identifies the user 430 using the second camera 413 and receives a user-customized parameter corresponding to the identified user 430 from the remotely located server 440. I can. The interfacing device 410 may access the server 440 using the Internet communication unit 415.

The control unit 416 may control the above-described operations. In particular, the control unit 416 may analyze modality inputs of the user 430 according to the user-customized parameter and generate new control information.

The interfacing device 410 may transmit the generated scene control information to the scene expression device 420. The scene expression device 420 may control an object included in the scene or a viewpoint of the scene itself, based on the scene control information.

At this time, the interfacing device 410 may generate control information necessary for the new expression device 420 to control the scene instead of continuously transmitting the sensing data or the recognition result to the new expression device 420 and transmit only the corresponding information. . Accordingly, the interfacing device 410 may provide a technology for optimizing the amount of data transmitted and received to provide a user interface.

Since matters described through FIGS. 1 to 3B may be applied to each of the modules shown in FIG. 4, a more detailed description will be omitted.

5 is a flowchart illustrating an interfacing method for providing a user interface utilizing multi-modality according to an exemplary embodiment.

Referring to FIG. 5, the interfacing method according to an embodiment may provide a user interface utilizing multi-modality.

For example, in the interfacing method according to an embodiment, in step 510, a scene parameter including information related to a scene may be obtained. The interfacing method may recognize at least two modality inputs input from a user in step 520. The interfacing method may generate scene control information based on a scene parameter and at least two modality inputs in step 540.

In some cases, the interfacing method may acquire user-customized parameters in step 530. In this case, the interfacing method may interpret at least two modality inputs input from the user based on the user customization parameter in step 540.

Since the matters described through FIGS. 1 to 4 may be applied to each of the steps shown in FIG. 5 as they are, a more detailed description will be omitted.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (17)

In an interfacing device that provides a user interface utilizing multi-modality,
To identify a user and obtain a scene parameter including information related to a characteristic of a scene capable of interacting with the user, and that the scene parameter corresponds to at least two modality inputs. Acquires a user-customized parameter indicating, and generates scene control information by analyzing at least two modality inputs based on the scene parameter and the user-customized parameter, and based on the scene control information, the At least one processor that selects at least one object included in the scene and controls transformation
Interfacing device comprising a.
The method of claim 1,
The scene parameter is
A parameter related to the size of at least one object included in the scene;
A parameter related to the location of the at least one object;
A parameter related to movement control of the at least one object;
A parameter related to rotation control of the at least one object;
A parameter related to scale control of the at least one object; And
Parameters related to the viewpoint of the god
Interfacing device comprising at least one of. The method of claim 1,
The processor is
Recognizing the user's gaze, the user's fingertips, the user's hand posture, the user's hand motion, or the user's voice
Interfacing device. The method of claim 1,
The scene control information is
A location element including information related to a specific location in the scene;
A transformation element including information related to transformation of the object located at the specific position;
A selection element including information related to selection of an object located at the specific location;
A viewpoint element including information related to the viewpoint of the scene;
A user identification element comprising information related to an identifier identifying a user; And
Association point element including information related to a plurality of points associated with the specific location
Interfacing device comprising at least one of. The method of claim 1,
The processor is
Interpreting a combination of the at least two modality inputs based on the user customization parameter
Interfacing device.
The method of claim 5,
The user-customized parameter is
Mapping information for mapping each of a plurality of predetermined control aspects of a plurality of scenes in response to a specific user to any one of a plurality of modality inputs or a combination of two or more selectable from the plurality of modality inputs according to the preference of the specific user
Interfacing device comprising a. The method of claim 5,
The processor is
Identify a user, and the user-customized parameter corresponds to the identified user
Interfacing device.
The method of claim 1,
The processor is
Receiving sensor data input through a sensor, and recognizing the at least two modality inputs based on the received sensor data
Interfacing device.
The method of claim 1,
The processor is
Communicates with a scene representation device representing the scene, receives the scene parameter from the scene representation device using a communication device, and transmits the generated scene control information to the scene representation device using the communication device doing
Interfacing device.
The method of claim 9,
The processor is
Controlling the scene based on the scene control information, and expressing the controlled scene
Interfacing device.
In the interfacing method that provides a user interface utilizing multi-modality,
Identifying a user;
Acquiring a scene parameter including information related to a characteristic of a scene capable of interacting with the user;
Obtaining a user-customized parameter indicating that the new parameter corresponds to at least two modality inputs;
Generating scene control information by analyzing at least two modality inputs based on the scene parameter and the user customized parameter; And
Selecting at least one object included in the scene and controlling transformation based on the scene control information
Interfacing method comprising a.
The method of claim 11,
The step of generating the scene control information
Interpreting the at least two modality inputs based on the user customization parameter; And
Generating scene control information based on the analyzed modality inputs and the scene parameter
Interfacing method comprising a.
The method of claim 11,
The user-customized parameter is
Mapping information for mapping each of a plurality of predetermined control aspects of a plurality of scenes in response to a specific user to any one of a plurality of modality inputs or a combination of two or more selectable from the plurality of modality inputs according to the preference of the specific user
Interfacing method comprising a.
The method of claim 11,
The scene control information is
A location element including information related to a specific location in the scene;
A transformation element including information related to transformation of the object located at the specific position;
A selection element including information related to selection of an object located at the specific location;
A viewpoint element including information related to the viewpoint of the scene;
A user identification element comprising information related to an identifier identifying a user; And
Association point element including information related to a plurality of points associated with the specific location
Interfacing method comprising at least one of. The method of claim 11,
Recognizing the at least two modality inputs
Recognizing the user's gaze;
Recognizing the user's fingertips;
Recognizing the user's hand posture;
Recognizing a user's hand motion; And
Step of recognizing the user's voice
Interfacing method comprising at least one of. The method of claim 11,
The scene parameter is
A parameter related to the size of at least one object included in the scene;
A parameter related to the location of the at least one object;
A parameter related to movement control of the at least one object;
A parameter related to rotation control of the at least one object;
A parameter related to scale control of the at least one object; And
Parameters related to the viewpoint of the god
Interfacing method comprising at least one of. A computer-readable recording medium on which a program for executing the method of any one of claims 11 to 16 is recorded.

Images